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A  User  Modelling  Approach  for  computer-based  Critiquing 
Thesis  directed  by  Professor  Gerhard  Fischer 

Tlieoietical  studies  and  implementations  of  computer-based  critiquing 
systems  indicate  that  it  is  desirable  fo  enhance  that  approach  to  better  support 
human-computer  collaborative  effort.  A  user  model  will  enable  these  systems  to 
individualize  explanations  of  their  advice  to  provide  better  support  for  cooperative 
problem  solving  and  enhance  user  learning.  User  modeUing  research  in  advice¬ 
giving  dialog  and  intelligent  educational  systems  was  studied  together  with 
theoretical  analyses  of  the  limitations  of  human-computer  interaction,  and  empiri¬ 
cal  observations  of  human-to-human  collaborative  effort.  A  framework  for  a  user 
modelling  component  for  a  critiquing  system  was  developed  and  implemented  in  a 
critic  for  LISP  programs.  The  user  models  developed  by  the  system  were  com¬ 
pared  to  self-assessment  questionnaires  completed  by  subjects  learning  the  LISP 
language.  The  analyses  indicated  a  favorable  correlation  and  potential  improve¬ 
ments  to  the  framework.  The  user  model  is  based  on  the  conceptual  domain  model 
required  for  explanations;  its  semantic  structure  allows  the  system  to  implicidy  en¬ 
rich  the  user  model  contents.  The  significance  of  this- weric  is  a  framework  for  a 
user  modelling  component  that  can  be  used  for  a  more  general  class  of  cooperative 
knowledge-based  systems.  Additionally,  using  the  structure  of  the  conceptual 
domain  model  as  the  basis  for  the  indirect  implicit  inference  techniques  is  unique. 
The  theoretical  foundations  for  the  work,  the  framework  developed,  and  an 
analysis  of  the  implementation  are  presented. 
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CHAPTER  I 


INTRODUCTION  AND  CONTEXT  FOR  THE  RESEARCH 

1.1.  Introduction  and  Overview 

This  thesis  discusses  a  user  modelling  approach  to  support  cooperative 
problem  solving.  The  problem  investigated  in  this  project  is  how  to  represent, 
acquire  and  provide  access  to  individual  user  models  to  support  computer  critics. 
Critics  are  knowledge-based  computer  systems  that  use  the  critiquing  approach  to 
support  their  users  in  their  work.  The  critiquing  approach  theoretically  enhances 
the  work  produced  by  their  users  (these  are  called  performance  critics)  and  sup¬ 
ports  their  learning  (called  educational  critics)  [Fischer  et  aL  90].  Future  systems 
that  support  users  in  their  working  envirorunents  need  the  usability  to  accomplish 
both  objectives.  A  user  model  will  be  an  important  component  of  such  a  system;  it 
will  assist  the  system  give  knowledge-based  advice  and,  when  it  is  appropriate, 
explain  that  advice.  A  framework  for  a  user  model  to  accomplish  this  was 
developed,  implemented,  and  evaluated  during  the  course  of  this  research.  Other 
user  modelling  research  was  studied  as  were  approaches  for  generating  explana¬ 
tions  of  domain  expertise.  Proven  techniques  from  other  user  modelling  research 
were  incorporated,  where  possible,  into  the  user  model.  An  understanding  of  what 
is  required  to  generate  explanations  guided  the  development  of  the  architecture  for 
the  user  modelling  component.  The  user  modelling  component  is  as  an  extension 
of  an  existing  critic  for  program  enhancement  (LlSP-CRmc). 

The  approach  to  user  modelling  combines  methodologies  developed  by 
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other  research  with  innovative  acquisition  techniques.  This  woik  is  unique  in  that 
it  investigates  enhancing  the  critiquing  paradigm  with  the  usability  to  individual¬ 
ize  the  explanations  of  advice  given  by  the  computer  critic.  The  major  contribu¬ 
tions  in  this  project  are  a  framework  for  a  user  modelling  component  for  critics 
that  is  also  of  potential  use  in  other  applications,  and  a  set  of  techniques  for  in¬ 
direct  implicit  acquisition  of  the  user  model.  These  techniques  use  the  semantic 
structure  of  the  conceptual  domain  model,  the  same  model  required  for 
explanation-giving.  The  frameworic  can  provide  support  to  both  a  broader  range 
of  q>plications,  and  to  systems  that  use  different  interaction  paradigms  (such  as 
tutoring  or  advising).  The  user  model  captures  the  expertise  of  individual  users  at 
the  conceptual  level  of  the  application  domain.  The  model  is  a  fine  grained 
representation  of  users’  knowledge  and  therefore  its  contents  could  support  other 
types  of  human-computer  interaaion.  Cooperative  problem  solving  systems  are  a 
general  class  of  knowledge-based  systems  that  will  help  friture  users  of  computer 
technology  to  both  accomplish  their  vocational  tasks  and  to  enhance  their  under¬ 
standing  of  the  application  domain.  The  approach  to  user  modelling  described 
here  is  general  enough  to  serve  the  general  class  of  cooperative  problem  solving 
systems. 

The  thesis  is  organized  into  three  sections.  Part  1  consists  of  Chapters  1 
through  4;  it  describes  the  author’s  understanding  of  the  theoretical  foundations 
and  analyzes  related  work.  The  four  areas  discussed  are:  using  computers  to  sup¬ 
port  cooperative  problem  solving  and  learning,  the  paradigm  of  critiquing,  an 
analysis  of  related  user  modelling  research,  and  the  implementation  environment, 
USP-CrtuC.  Part  2  consists  of  Chapters  5  through  7;  it  covers  the  instantiation  of 
the  user  modelling  approach  in  system  design  and  implementation.  A  requisite 
domain  model  to  support  this  work,  a  framework  for  an  explanation  component 
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that  makes  use  of  the  user  model,  and  the  user  modelling  component  that  was 
developed,  are  described.  Part  3,  Chapters  8  through  10,  analyzes  the  effective¬ 
ness  of  the  implementation  and  the  contributions  of  the  woik.  Possible  directions 
for  continuing  the  research  are  identified,  and  the  thesis  concludes  with  a  sum¬ 
mary. 

When  people  use  a  knowledge-based  system  they  expect  that  it  will  help 
them  to  produce  a  better  product.^  As  a  by-product  of  this  process,  users  improve 
their  understanding  of  the  application  domain  for  that  product.  Therefore,  to  im- 
derstand  what  it  means  for  any  system  to  support  both  doing  and  learning,  it  was 
necessary  to  examine  two  paradigms: 

•  cooperative  problem  solving  systems,  and 

•  user-centered  computer  learning  environments. 

The  ability  to  explain  its  actions  is  a  necessary  characteristic  for  a  system  that  at¬ 
tempts  cooperative  interaction  and  also  supports  learning:  To  achieve  this  it  is 
necessary  for  these  collaborative  systems  to  employ  user  models  to  individualize 
those  explanations. 

Creating  computer  systems  that  facilitate  cooperation  between  a  human 
and  a  computer  requires  more  than  just  developing  powerful  interaction  tech¬ 
nologies.  We  need  an  approach  to  computer  support  of  problem  solving  that  in¬ 
cludes  knowledge-based  techniques,  a  computer-user  dialog  based  on  the  idea  of 
natural  communications,  and  support  for  system  adaptivity.  The  types  of  systems 
that  achieve  this  will  be  collaborative  symbiotic  human-computer  working  en¬ 
vironments  that  support  user  learning.  A  conceptual  framework  for  systems  that 
use  knowledge-based  techniques  to  aid  users  in  accomplishing  their  tasks  is 
provided  by  the  cooperative  problem  solving  paradigm. 

^Product  is  used  here  in  a  general  sense;  it  includes  both  specific  objects 
generated  by  the  work,  such  as  a  design,  and  abstract  results,  such  as  a  decision. 
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1.2.  Cooperative  Problem  Solving 

There  are  methods  and  technologies  in  the  field  of  Artificial  Intelligence 
that  can  help  improve  the  productivity  of  con^uter  users.  A  paradigm  for  design¬ 
ing  systems  that  goes  beyond  current  autonomous  expert  systems  to  address 
human  needs  and  potential  is  that  of  Cooperative  Problem  Solving  Systems 
[Fischer  90].  These  use  knowledge-based  techniques  to  work  in  symbiotic  con¬ 
sonance  with  the  user.  The  systems  are  cooperative  in  that  they  operate  in  a 
similar  marmer  to  the  way  a  helpful  person  acts,  and  they  attempt  to  assist  their 
users  as  best  they  can.  The  relationship  between  the  human  and  the  con^uter  is 
symbiotic  in  that  there  is  mutual  benefit;  the  resulting  product  of  their  collabora¬ 
tion  is  better  than  either  could  produce  by  themselves. 

In  cooperative  problem  solving,  the  user  and  computer-based  system 
work  on  the  same  problem  using  a  collaborative  interaction  style.  Systems  that 
can  support  cooperative  problem  solving- will  have  to  fit  Ulich’s  pUich  73]  notion 
of  convivial  tools  [Fischer  83].  Cooperative  systems  allow  the  combining  of 
human  skills  and  computing  power  to  accomplish  a  task  which  could  not  be  done 
by  either  the  human  or  the  computer  alone;  or  in  those  cases  where  it  could  be,  the 
quality  of  the  result  or  the  speed  with  which  a  solution  is  obtained  is  significantly 
improved  when  the  two  agents  work  together.  The  idea  of  symbiotic  cooperation 
between  the  user  and  the  computer  has  also  been  embraced  by  researchers  in  the 
related  field  of  decision  support  systems  [Mili,  Manheim  88;  Manheim,  Srivas- 
tava,  Vlahos,  Hsu,  Jones  90;  Hefley  90].  To  achieve  a  cooperative  system  requires 
some  degree  of  adaptivity  to  individual  users;  a  system  can  adtqrt  successfully 
when  it  knows  something  about  the  individual;  and  user  modelling  provides  to  a 
system  the  capability  to  acquire  and  use  that  type  of  knowledge. 
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The  objective  of  an  interaction  between  a  human  and  a  computer 
generally  falls  into  one  of  two  classes:  problem  solving  or  information  retrieval.^ 
Design  might  be  considered  as  a  third  major  class;  the  view  here  is  that  design  is  a 
subset  of  problem  solving  because  the  design  is  generated  to  address  problematic 
needs  of  either  users  themselves  or  those  of  a  client  for  whom  they  are  working. 
Information  retrieval  is  often  one  part  of  problem  solving.  The  computer’s  role  is 
to  aid  users  in  arriving  at  a  solution  or  to  help  locate  information  that  solves  or 
helps  to  solve  their  problems. 

Autonomous  expert  systems  have  a  different  ^proach  —  they  develop 
problem  solutions  independent  of  user  input,  except  for  the  problem  specification 
or  task  designation.  In  many  problem  solving  and  information  retrieval  situations, 
articulation  of  the  task  is  difficult  and  people  are  unsure  of  their  objective  or  exact 
problem.  They  start  with  a  general  view  of  what  they  expect  to  achieve  and  refine 
their  own  understanding  and  problem  specification  as  part  of  the  solution  process. 
To  put  cooperative  problem  solving  systems  into  the  context  of  current  artificial 
intelligence  technology  we  need  to  consider  system  designs  that  go  beyond  current 
notions  of  expert  systems  and  to  understand  when  it  is  appropriate  to  use  these 
alternative  approaches. 

Feigenbaum  and  McCorduck  state  in  their  book  on  expert  systems: 
’’Most  knowledge-based  systems  are  intended  to  assist  human  endeavor  and  are 
almost  never  intended  to  be  autonomous  agents”  [Feigenbaum,  McCorduck  83,  p. 
115].  This  view,  unfortunately,  is  not  held  throughout  the  field.  Those  expert 
systems  often  cited  as  the  major  success  stories  of  the  past  10  to  15  years,  for 

^Amusement  is  also  a  significant  {plication  for  computers,  but  we  focus  on 
applying  technology  in  the  workplace  to  accomplish  a  specific  goal  or  task,  rather 
than  using  it  to  entertain. 
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example  Rl,  MYCIN,  Dipmeter  Advisor,  have  been  designed  as  domain  experts  that 
are  ct^able  of  solving  a  certain  set  of  problems.  These  are  problems  that  require 
either  the  heuristic  knowledge  of  a  human  expert  with  considerable  experience  or 
problems  that  require  excessive  computation  to  yield  to  timely  solutions  using 
standard  algorithmic  ^roaches. 

The  major  difference  between  classical  expert  systems  (such  as  Rl  or 
MYON)  and  cooperative  problem  solving  systems  is  that  the  users  of  cooperative 
problem  solving  systems  are  active  agents.  They  are  actively  engaged  in  reason¬ 
ing  about  the  problem  and  generating  the  solution  rather  than  participating  as  mere 
providers  of  information  to  the  system.  Conversely,  traditional  expert  systems  ask 
users  for  information  about  the  problem  situation  and  then  return  a  solution;  from 
an  operational  perspective  they  appear  as  a  “black  box.”  Cooperative  problem 
solving  system  are  designed  so  that  the  user  and  the  system  share  in  the  problem 
solving  and  in  the  decision  making.  Because  human-computer  communication  is 
central,  cooperative  problem  solving  systems  require  better  interaction  facilities 
than  those  offered  by  traditional  expert  systems. 

When  knowledge-based  systems  support  decision  at  higher  levels  of 
societal  and  organizational  responsibility  they  should  not  usurp  the  user’s  respon¬ 
sibilities  for  a  decision.  For  example,  a  commander  of  a  military  operation  should 
have  access  to  the  expertise  ctq>tured  in  a  knowledge-based  system  that  knows 
strategy,  combat  resources,  and  heuristics  about  how  to  apply  them  to  accomplish 
the  mission.  However,  this  situation  involves  significant  danger  to  hiunan  lives 
and  we  would  not  want  to  turn  control  completely  over  to  an  expert  system  to  run 
the  battle.  Similar  scenarios  exists  in  natural  disaster  emergency  plaiming,  and  the 
operating  of  complex,  potentially  dangerous  equipment  (e.g.,  nuclear  power  sta¬ 
tions).  Autonomous  expert  system  approaches  must  be  replaced  by  a  more  general 
knowledge-based  system  paradigm,  such  as  cooperative  problem  solving  systems. 


Commimicadons  between  a  human  and  a  computer  is  a  fimdamental 
design  problem  for  cooperative  systems.  Specifically,  to  facilitate  interaction  be¬ 
tween  a  human  and  a  computer  it  is  necessary  to  exploit  the  asymmetry  of  the  two 
communication  partners.  Each  agent  or  parmer  contributes  what  they  can  do  best. 
People  are  better  than  computers  at  applying  common  sense,  defining  goals,  and 
decomposing  problems.  Computers  should  be  used  as  an  external  memory,  to  do 
consistency  checking,  to  hide  but  not  lose  irrelevant  information,  to  capture  and 
summarize  problem  solution  steps,  and  to  help  visualize  concepts.  In  this  diesis 
the  user  model  serves  a  system  in  which  the  user  is  a  programmer  who  understands 
the  problem  and  develops  the  code  to  solve  that  problem.  The  computer  does  not 
understand  the  problem  and  could  not  write  a  program  (automatic  program  genera¬ 
tion)  to  solve  it  even  if  it  did.  But  the  system  knows  more  efficient  ways  to  imple¬ 
ment  program  code.  The  system  will  be  described  in  detail  in  Chapter  4  but  in  the 
context  here  it  is  important  to  see  that  the  system  is  designed  to  exploit  each 
agent’s  expertise  by  sharing  responsibility  for  producing  a  good  solution.  The 
programmer  produces  code  that  algorithmically  solves  the  problem  and  the  system 
reviews  that  code  suggesting  to  the  programmer  ways  to  improve  it. 

Communicating  means  an  agent,  human  or  computer,  has  to  know  or 
must  assume  something  about  its  partner.  One  approach  is  for  systems  to  capture 
implicit  assumptions  about  all  of  the  users  in  their  design  (a  default  generic  user 
model).  Ideal  systems  might  adapt  everything  they  do  to  each  individual.  A  more 
reasonable  middle  ground,  is  to  have  systems  tailor  their  side  of  an  interaction 
based  on  what  they  are  able  to  infer  about  their  human  parmer  by  applying  user 
modelling  methods.  Understanding  what  is  required  to  accomplish  user  modelling 
requires  an  examination  of  the  human-computer  communication  process. 
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When  two  agents  aie  engaged  in  cooperative  effort,  a  process  of  natural 
communication  takes  place.  Natural  communication  is  more  than  natural  lan¬ 
guage;  it  is  the  ability  to  engage  in  a  dialog  that  makes  use  of  diectic  techniques, 
indexicals,  graphic  representation,  and  references  to  previous  conversation.  In 
human  dyads  more  goes  on  than  an  exchange  of  information,  if  one  partner  serves 
as  an  advisor  or  critic  they  are  expected  to  imderstand  what  the  other  is  trying  to 
do  and  guide  them  correctly.  Cooperative  computer  systems  need  techniques  for 
helping  them  attempt  similar  efforts.  There  is  significant  research  into  techniques 
for  goal  and  plan  recognition  on  the  part  of  computers  and  also  considerable  skep¬ 
ticism  whether  computers  will  ever  be  able  to  accomplish  this  [Suchman  87; 
Dreyfus,  Dreyfus  86].  This  does  not  restrict  systems  from  using  knowledge  of 
goals  or  plans  that  can  be  obtained  by  querying  the  users  to  help  stmcture  inter¬ 
actions  with  them.  The  user  model  component  plays  a  role  in  that  it  must  be  able 
to  represent  users’  goals  and  provide  tharinformation  to  the  system  aid  in  the  com¬ 
munication  process.  Obtaining  that  information  indirectly  may  eventually  be  part 
of  acquiring  the  user  model,  but  it  is  going  to  require  effective  goal^lan  recog¬ 
nition  techniques  that  are  the  subject  of  another  direction  of  research  in  artificial 
intelligence. 

It  is  important  to  think  in  terms  of  natural  communications  so  that  tech¬ 
niques  in  addition  to  natural  language  are  integrated  into  system  designs.  Systems 
that  use  natural  language  generation  and  recognition  techniques  are  frequently  too 
brittle  [Winograd,  Flores  86].  They  experience  breakdowns  in  interacting  with 
users  when  the  unexpected  occurs,  especially  in  situations  not  anticipated  by  the 
system  designer;  techniques  are  needed  to  get  past  the  breakdown.  Another  reason 
to  think  about  the  entire  spectrum  of  communication  techniques  is  that  techniques 
for  natural  language  generation  and  interpretation  have  matured  to  the  point  where 
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they  are  generally  useful  in  systems  such  as  LlSP-CRmc.  As  is  discussed  in  Chap¬ 
ter  4,  the  communication  medium  for  LiSP-CRmc  is  a  set  of  available  techniques 
provided  in  a  powerful  workstation  environment  (menus,  command  languages,  and 
hypertext).  While  waiting  for  natural  language  capabilities  to  mature  to  the  point 
of  general  utility,  it  is  necessary  to  build  systems  that  address  real  needs  using 
available  technology,  and  to  use  them  as  a  context  for  related  research,  such  as 
user  modelling. 

Related  to  natural  language  limitations  is  the  idea  of  situated  action 
[Suchman  87].  Research  by  Suchman  using  the  situated  action  perspective  high¬ 
lights  some  inevitable  shortcomings  of  human-computer  interaction  paradigms. 
The  limited  bandwidth  across  which  the  human  and  computer  can  communicate 
preclude  the  machine  from  having  access  to  both  the  quantity  and  quality  of  infor¬ 
mation  available  to  a  human.  This  observation  should  motivate  efforts  that  inves¬ 
tigate  how  to  improve  the  capabilities  of  computers.  User  modelling  is  one  tech¬ 
nique  that  can  help  the  computer  to  improve  on  its  ability  to  aid  a  user  in  the 
situated  context  The  degree  to  which  it  will  help  is  an  open  research  issue  that 
can  only  be  investigated  after  suitable  user  modelling  techniques  are  developed 
and  in^lemented. 

Computers  can  understand  task  domain  knowledge;  this  knowledge  can 
be  used  as  the  basis  for  advice,  and  as  a  source  for  guidance  about  how  to  better 
communicate  that  advice  and  explain  it.  The  distinction  between  advising  users 
(telling  them  what  to  do)  and  explaining  that  advice  (telling  them  the  reason  for  it) 
is  not  made  in  most  research.  A  notable  exception  is  the  work  on  the  EUROHELP 
project  [Kelleher  88].  In  this  thesis  the  distinction  is  significant  because  the  sub¬ 
ject  system  gives  both  suggestions  (or  advice)  and  we  want  to  endow  it  with  the 
ability  to  explain  those  suggestions  to  the  user.  It  is  the  latter  process  that  makes 
use  of  the  content  of  the  user  model. 
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A  model  of  the  individual  user  can  be  an  important  component  of  any 
system,  such  a  model  can  aid  in  the  natural  communication  process,  assist  in 
managing  breakdowns,  and  help  make  systems  more  acceptable  to  their  users. 
Modelling  anodier  agent  occurs  on  both  sides  of  a  cooperative  dyad.  Users  devel¬ 
op  models  of  the  systems  with  which  they  interact  and  computers  need  to  be 
designed  so  they  can  develop  models  of  htiman  users.  The  work  presented  here 
focuses  on  the  latter  class  of  models. 

A  user  modelling  component  will  not  solve  all  human-computer  inter¬ 
action  problems  but  it  is  essential  to  investigate  its  potential  impact.  We  must  j5rst 
develop  system  techniques  for  representing  and  building  these  models.  But  even 
with  good  user  models,  cooperative  problem  solving  systems  will  not  always  be 
successful  in  their  initial  attempt  to  provide  advice  or  explanation  —  what  is 
sometimes  called  a  “one-shot”  approach  to  the  interaction.  Even  people  do  not 
always  “get  it  right”  the  first  time;  a  great  deal  of  the  effort  in  any  communica¬ 
tions  involves  repairing  breakdowns  between  the  two  partners.  Techniques  to 
achieve  something  similar  are  needed  in  interactive  systems.  When  users  do  not 
understand  the  system’s  advice,  critique,  or  explanation,  followup  techniques  are 
required  [Moore  89]. 

Human  experts  model  their  communications  partner  in  order  to  provide 
the  appropriate  level  of  assistance  and  explanation.  This  motivates  the  require¬ 
ment  for  cooperative  computer  systems  to  be  designed  to  attempt  something 
similar.  Reeves  conducted  an  empirical  study  of  collaborative  problem  solving 
efforts  where  sales  clerks  in  a  large,  well-stocked  hardware  store  assist  patrons  in 
solving  their  problems  [Reeves  90].  When  interviewed,  these  expert  agents  related 
that  part  of  the  process  involved  modelling  the  client  so  that  the  advice  given  and 
any  explanations  requested  could  be  specifically  designed  for  that  individual. 
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Modelling  their  customers  played  a  crucial  role  in  identifying,  for  these  sales 
agents,  the  level  at  which  to  share  their  understanding. 

Communication  between  cooperating  agents  can  be  viewed  in  terms  of 
two  roles,  that  of  speaker  and  listener.  The  speaker  presents  information  and  the 
listener  interprets  it.  The  listener’s  role  is  usually  more  difficult  because  the  lis¬ 
tener  has  to  understand  the  problem  based  on  the  speaker’s  description. 
Knowledge-based  systems  that  communicate  with  a  user  have  to  be  designed  to 
accommodate  both  roles.  This  is  especially  true  in  cooperative  problem  solving 
systems  where  users  play  an  active  role  in  both  the  problem  solving  and  the  deci¬ 
sion  making  processes.  In  this  research  the  interface  confined  users  to  com¬ 
municating  with  the  system  using  only  available  technologies;  a  natural  language 
context  was  not  assumed.  This  restriction  on  users  is  necessary  in  order  to  accom¬ 
modate  the  system’s  role  as  a  listener.  The  system  serves  in  the  speaker  role  when 
it  gives  suggestions  and  when  it  explains  suggestions.  The  user  model  developed 
here  was  primarily  designed  to  accommodate  the  system  in  the  speaker  role.  Ul¬ 
timately,  user  models  will  have  to  help  the  system  fulfill  both  roles. 

In  addition  to  helping  users  to  solve  their  problems,  systems  should  also 
help  them  learn  about  the  task  domain.  Systems  that  serve  knowledge  workers, 
such  as  designers,  authors,  and  programmers,  must  accomplish  both  objectives.  In 
normal  use  it  is  difficult  to  distinguish  a  situation  or  episode  oriented  strictly  on 
problem  solving;  learning  and  doing  fiequently  intermix  during  human-computer 
interactions.  Support  for  user  learning  is  also  a  goal  of  a  cooperative  knowledge- 
based  system,  so  the  theory  behind  computer  learning  environments  was  also  ex¬ 
amined. 


M. 


13.  Learning  Environments 


Computer-based  learning  environment  enable  users  to  improve  their 
proficiency  in  a  domain  by  providing  for  the  knowledge  communication  process 
[Wenger  87].  One  ^roach  to  designing  such  learning  environments  is  tihat  of 
building  instructionally  focused  systems,  such  as  intelligent  tutors  [Sleeman, 
Brown  82],  In  many  situations  it  is  desirable  to  provide  a  learning  opportunity 
within  the  context  of  a  user’s  work  and  with  the  user  in  control  of  the  interaction. 
Here,  paradigms  that  are  more  general  than  intelligent  tutoring  systems  are  needed. 
There  is  interest  and  research  that  addresses  the  problems  in  developing  learning 
environments  in  several  disciplines  to  include  cognitive  science  [Burton,  Brown, 
Fischer  84;  Fox  88],  human-computer  interaction  [Fischer  88a],  and  computer- 
based  training  technology  [Duchastel  88;  Mastaglio  90a]. 

The  opportunity  to  develop  computer-supported  learning  environments 
has  motivated  the  reexamination  of  existing  paradigms  of  education  and  the  for¬ 
mulation  of  new  ones.  Many  of  these  paradigms  connect  learning  with  experience, 
ascribing  to  a  philosophical  view  and  emphasizing  techniques  that  make  this  con¬ 
nection  the  basis  for  system  design  [Psotka,  Massey,  Mutter  88a].  Because  this  is 
an  emerging  discipline  the  learning  approaches  have  various  names,  such  as:  col¬ 
laborative  learning,  reactive  learning,  situated  learning,  learning  on  demand,  and 
incremental  learning.  There  is  some  overlap  between  the  paradigms;  what  is  sig¬ 
nificant  is  that  they  share  a  common  ideal.  Their  goal  is  to  provide  opportunities 
to  leam  skills  by  practicing  them  in  a  realistic  work  setting  —  the  workplace  or 
system  in  which  the  .skills  are  to  be  used,  or  a  simulation  of  that  environment.  The 
tqjproach  of  augmenting  a  work  environment  is  weU  suited  to  situations  where  the 
computer  system  is  that  enviroiunent  (for  example  a  CAD/CAM  system).  Com¬ 
puters  are  good  for  simulating  in  circumstances  where  training  on  the  actual  equip- 
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ment  (for  example  a  power  plant  or  aircraft)  is  prohibitively  expensive,  or  there  are 
safety  considerations.  In  order  to  individualize  learning  opportunities,  systems 
need  to  know  what  knowledge  their  users  have  about  the  domain;  idiosyncratic 
models  of  individual  domain  expertise  are  needed.  To  understand  what  it  means  to 
have  such  a  model,  the  theories  behind  learning  approaches  for  interactive  en¬ 
vironments  was  examined. 

13.1.  Foundations  for  Learning  Environments 

There  is  a  need  to  find  efficient  and  practical  ways  to  improve  education 
using  computer  technology.  Studies  by  Bloom  and  colleagues  demonstrated  that  if 
we  can  individualize  instruction,  significant  increases  in  student  performance  can 
be  reasonably  expected  [Bloom  84].  The  computer  can  provide  an  efficient 
methodology  for  individualized  learning  situations  and  researchers  concerned  with 
education  and  training  want  to  use  computer-based  training  to  meet  educational 
needs  in  a  variety  of  contexts  [Seidel,  Weddle  87]. 

A  shortcoming  of  early  computer  learning  systems  was  their  fundamen¬ 
tal  design  philosophy;  it  was  based  on  conventional  ideas  about  programmed  in¬ 
struction  used  for  self-paced  learning  material.  An  alternative  approach  using  ar¬ 
tificial  intelligence  techniques  was  first  proposed  by  Carbonell  [Carbonell  70]. 
During  the  ensuing  20  .>  ■  since  that  work,  research  efforts  have  resulted  in  the 

development  of  sev  ^  paradigms  for  computer-based  instruction  using  artificial 
intelligence  [Wenger  87].  The  computer  can  provide  a  suitable  context  for  learn¬ 
ing  both  procedural  and  declarative  knowledge  and  as  reasonably  priced  hardware 
support  for  graphical  displays  becomes  a  reality,  these  types  of  environments  will 
become  more  feasible.  The  theoretical  limitation  to  effectiveness  is  not  the 
availability  of  material  or  simulated  scenarios  but  incorporating,  into  the  system,  a 
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didactic  agent  to  guide  learners.  For  learning  to  occur  in  a  computational  context 
in  which  users  are  involved  in  some  action,  the  system  must  provide  feedback  to 
its  users  in  the  form  of  advice,  critiques,  and  explanations;  higher  quality  of  the 
feedback  results  in  an  improved  learning  process. 

One  way  to  provide  a  system  the  ability  to  provide  high  quality  feedback 
is  using  knowledge-based  components.  Three  processes  in  the  learning 
environments  require  the  use  of  knowledge: 

1.  providing  information  or  instruction  with  an  expected  outcome  that 
users’  knowledge  improves, 

2.  determining  the  state  of  users’  present  knowledge  (user  or  student 
modelling),  and 

3.  motivating  the  user  to  learn. 

To  execute  the  first  process,  a  computer  agent  has  to  know  both  the  domain  and 
strategies  for  guiding  users  in  the  direction  of  learning.  Methods  to  accorr^lish  the 
second  process  are  the  subject  of  this  research.  Cooperative  problem  solving 
systems  assume  that  the  third  process  is  inherent  in  the  situation;  they  assume  that 
users  are  motivated  because  they  have  chosen  to  use  the  system  in  the  first  place. 

After  investigating  the  literature,  I  found  there  are  three  distinguishable 
paradigms  for  computer-based  education  based  on  artificial  intelligence  techniques 
—  three  different  classes  of  intelligent  learning  environments  [Mastaglio  90a]: 

•  Intelligent  tutoring  systems  present  instructional  material  in  a  manner 
similar  to  a  classroom  teacher.  [Sleeman,  Brown  82;  Psotka,  Massey, 
Mutter  88b;  Poison,  Richardson  88]. 

•  The  coaching  approach  does  not  use  teaching  techniques  in  the  stric¬ 
test  sense  of  the  lecturing  classroom  teacher  but  its  metaphorical  basis 
is  the  human  coach  who  places  students  into  a  suitable  context,  then 


observes  how  they  perform  and  provides  advice  on  how  to  improve 
[Burton,  Brown  82]. 

•  The  paradigm  that  serves  as  the  foundational  interaction  methodology 
in  this  work  is  the  computer-based  critic  [Fischer,  Lemke,  Mastaglio, 
Morch  90].  It  serves  as  an  intelligent  agent,  able  to  evaluate  user’s 
work  on  call  like  a  human  mentor  or  colleague.  The  critic  is  a  domain 
expert  ready  to  evaluate  users’  actions  and  provide  suggestions  when¬ 
ever  asked. 

In  this  research,  critiquing  had  been  emphasized  over  tutoring  because  it 
holds  promise  as  a  more  general  approach.  Education  and  training  should  not  be 
viewed  as  only  isolated  activities  that  occur  during  set  periods  of  a  lifetime  where 
the  focus  is  on  the  acquisition  of  skills  and  knowledge.  Instead,  the  broader 
perspective  taken  is  that  they  are  life  long  activities  needed  to  maintain  proficiency 
and  accommodate  changes  in  domain  theory  or  technology.  A  computer-based 
critic  can  help  users  improve  their  skills  within  their  working  context,  but  besides 
giving  suggestions  they  need  to  be  able  explain  their  expertise,  not  just  in  a 
canonical  form  but  in  a  manner  that  is  tailored  to  each  user’s  current  state  of 
knowledge.  For  critics  to  fully  support  the  learning  process  they  will  require  ex¬ 
planation  and  user  modelling  capabilities. 

Critiquing  is  not  the  only  approach  to  designing  a  learning  environment 
but  it  is  effective  in  the  application  domain  (progranuning)  for  this  research.  The 
user  model  developed  should  be  able  to  serve  not  just  critiquing  but  the  general 
class  of  learning  environments.  To  achieve  that  goal  some  of  the  theoretical  foun¬ 
dations  for  learning  environments  were  studied.  The  notions  of  situated  learning 
[Brown,  Burton,  Kleer  82],  and  learning  on  demand  [Fischer  88a]  provided 
general  guidance  and  understanding. 


13.2.  Learning  on  Demand 

Learning  on  demand  supports  learning  in  the  context  of  a  user’s  work, 
allowing  people  to  improve  their  knowledge  whenever  a  need  arises  [Fischer  87a]. 
It  is  based  on  an  optimistic  view  that  people  want  to  know  how  to  do  their  jobs 
better  and  are  willing  to  engage  in  learning  activities.  If  computer-based  working 
environments,  (e.g.,  design  environments  [Lemke  89])  are  to  provide  a  con:q>lete 
and  appealing  context  for  working,  then  they  need  to  support  learning  on  demand. 
Learning  environments  can  be  designed  as  extensions  of  existing  computer-based 
systems,  ones  that  already  support  some  types  of  work.  Some  examples  of  these 
are  design,  programming,  and  authoring. 

The  need  to  support  learning  on  demand  requires  architectures  that  differ 
from  instmctionally  oriented  systems.  Learning  on  demand  requires  that  the  user 
retain  primary  control  of  the  learning  situatio  n.  A  continuum  of  approaches  to 
designing  learning  environments  is  shown  in  Figure  1-1;  the  dimension  for  the 
continuum  is  control  of  the  interaction.  At  the  left  extreme,  primary  control  is  the 
responsibility  of  the  system;  at  the  right,  the  responsibility  of  the  user.  In  the  cen¬ 
ter,  the  system  and  the  user  share  responsibility  for  control.  Exploratory  learning, 
in  the  spirit  of  LOGO  [Papert  80],  is  the  type  of  system  found  at  end  of  the  con¬ 
tinuum  where  the  user  has  total  control  of  the  interaction.  Exploratory  learning 
environments  do  not  require  domain  knowledge,  but  they  must  be  carefully 
designed  to  provide  opportunities  for  interesting  exploration  while  at  the  same 
time  protecting  users  from  fatal  errors.  Traditional  computer-aided  instruction 
(CAI)  is  shown  at  the  other  end  of  the  continuum.  These  systems  are  algorith¬ 
mically  controlled  by  the  computer  program  and  allow  minimal,  if  any,  student 
control;  traditional  CAI  systems  also  do  not  have  knowledge  in  the  sense  of  artiE- 
cial  intelligence. 
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Figure  1-1:  A  Continuum  of  Approaches  to  Learning  Environments 


In  the  middle  of  the  spectrum  are  the  three  knowledge-based  paradigms 
for  learning  environments  that  were  previously  enumerated.  Within  the  class  of 
intelligent  tutoring  systems  various  degrees  of  control  may  be  given  to  the  student 
but,  in  general,  problem  selection,  monitoring  of  student  actions,  and  intervention 
fall  under  the  auspices  of  the  intelligent  tutor.  Coaching,  as  used  in  the  WEST 
system  [Burton,  Brown  82],  allows  users  to  practice  sldUs  in  a  computer-supported 
context  with  a  computer-based  intelligent  mentor  “watching  over  their  shoulder.” 
The  coach  intervenes  with  suggestions  and  instruction  when  appropriate. 

Research  efforts  in  both  exploratory  learning  [Miller  79]  and  coaching 
systems  [Brown,  Burton,  Kleer  82]  recognized  the  need  for  a  paradigm  which  is 
not  as  intrusive  as  a  coach  but  extends  the  power  of  exploratory  envirorunents  by 
providing  the  user  contextual,  on  call,  intelligent  assistance  in  the  application 
domain.  Computer  critiquing  is  a  paradigm  which  meets  that  requirement,  it  al¬ 
lows  users  mote  control  of  the  interaction  as  well  as  responsibility  for  selecting  the 
problem. 

A  system  that  provides  for  learning  on  denumd  carmot  help  but  situate 
that  learning  in  the  user’s  context.  Because  it  is  in  this  context  that  users  will 
request  support  for  learning;  for  the  system  to  coerce  them  into  a  special  learning 
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microworld  defeats  the  objective  behind  learning  on  demand.  Realizing  the 
situatedness  of  the  working  and  learning  context  makes  it  important,  in  this 
research,  to  tmderstand  the  theoretical  studies  of  situated  action  and  consider  their 
impact  on  the  goals  of  the  user  modelling  effort. 

13.3.  Situated  Action 

Suchman  made  an  in-depth  study  of  how  people  interact  with  machines 
[Suchman  87]  and  argues  that  research  approaches  which  atten^t  to  explicitly 
represent  or  infer  user  plans  are  inadequate.  First  we  need  to  explore  the  relation¬ 
ship  of  knowledge  to  action,  keeping  in  mind  that  a  machine’s  resources  for  inter¬ 
preting  the  user’s  behavior  are  significantly  poorer  than  those  of  a  human. 

A  cautious  attitude  toward  what  to  expect  from  a  machine  is  important 
because  during  face-to-face  communication  between  people  there  are  resources 
that  help  them  detect  and  remedy  trouble  when  it  develops,  for  example,  facial 
expression  or  tone  of  voice,  among  others.  The  same  range  of  resources  are  lack¬ 
ing,  for  the  most  part,  in  human-machine  interaction  because  of  the  impoverished 
communication  charmel.  A  pessimistic  perspective  would  discount  attempts  to 
make  machines  act  intelligently.  A  more  optimistic  view  is  embraced  here,  the 
philosophy  that  it  is  important  to  investigate  whatever  possibilities  to  support  the 
system  do  exist,  while  keeping  the  limitations  of  situated  action  in  mind.  Research 
on  natural  language  as  a  communication  medium  assumes  understanding  and  ex¬ 
pression  ctq>abilities  on  the  part  of  the  computer.  The  orientation  in  this  project 
was  very  different,  the  work  looked  at  those  conununicadons  capabilities  which 
are  available  with  current  technology  to  determine  how  they  could  best  be  used  to 
improve  what  the  systems  knows  about  the  user,  and  also  what  knowledge  about 
the  domain  is  required  by  the  system  to  make  best  use  of  these  capabilities.  One 
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such  idea  considers  the  content  of  the  man-machine  interaction  dialog  as  a  source 
of  information  that  can  aid  in  acquiring  models  of  users. 

An  environment  that  provides  users  the  opportunity  to  learn  in  the  con¬ 
text  of  the  task  domain  or  by  using  a  simulation  of  that  context,  requires  that  the 
system  react  so  as  to  increase  their  understanding  of  the  situated  action  [Brown, 
Burton  86].  Critiquing  can  facilitate  situated  learning  because  it  provides  learning 
opportunities  in  the  work  context.  The  critique  exposes  situations  for  improving 
the  product  or  the  actions  of  users;  these  are  learning  opportunities  as  well.  If 
users  do  not  imderstand  the  critique  then  there  is  an  opportunity  for  an  explanation 
in  context.  They  learn  a  set  of  conditions  under  which  their  knowledge  can  be 
appUtd  and  as  a  result  improve  their  understanding.  This  learning  scenario  is  a 
task-driven  environment,  providing  a  unique,  situated  context  for  learning  to  take 
place.  This  context  is  the  central  notion  in  situated  learning;  in  critiquing  systems 
it  comes  about  naturally.  Critiquing  combined  with  explanation  approaches  can 
clarify  understanding  and  help  to  restructure  users’  knowledge  [Psotka,  Massey, 
Mutter  88a].  Learning  has  been  traditionally  supported  with  instruction,  but  a 
more  likely  situation,  one  similar  to  the  manner  in  which  human-to-human  inter¬ 
action  occurs,  is  to  support  it  with  an  explanation  usability  [Wenger  87].  Ex¬ 
planation  of  the  system’s  critique  is  stipported  by  the  user  modelling  system  that 
was  developed  in  this  research. 

1.4.  Summary 

This  chapter  examined  the  context  for  this  research  in  terms  of 
paradigms  for  cooperative  problem  solving  and  computer  systems  that  support 
learning.  The  two  cannot  be  easily  s^arated  on  either  a  theoretical  level  or  in 
systems  designed  to  support  users  in  their  work.  Therefore,  both  become  con- 
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siderations  in  how  systems  should  be  designed  and  in  detennining  the  role  played 
by  their  user  modelling  component.  To  achieve  true  cooperativity,  systems  must 
adapt  to  their  users;  while  support  for  learning  requires  that  they  provide  the  users 
feedback  and  explanations;  explanations  are  best  when  tailored  to  the  individual. 
This  means  idiosyncratic  models  of  users  are  required.  The  exact  content  of  those 
models  is  determined  by  the  needs  of  the  systems  that  will  use  the  models.  This 
cluq)ter  has  explained  the  theoretical  concepts  for  designing  cooperative  systems 
that  support  task  accomplishment,  and  the  motivation  for  having  systems  that  sup¬ 
port  learning;  both  of  these  provide  motivating  goals  for  this  dissertation  research. 
The  critiquing  paradigm,  the  specific  system  design  framework  within  which  the 
user  modelling  work  was  completed,  will  be  discussed  next. 


CHAPTER  n 


CRITIQUING 

Critiquing,  as  a  technique  for  building  systems,  is  of  research  interest  in 
both  artificial  intelligence  and  human-computer  interaction.  It  has  been  a  major 
topic  of  investigation  for  the  Human-computer  Communications  Group  at  the 
University  of  Colorado  during  the  past  several  years  [Fischer,  Mastaglio  89;  Fis¬ 
cher  et  al.  90;  Fischer,  Mastaglio  90;  Fischer,  Lemke,  Mastaglio,  Morch  90;  Mas¬ 
taglio  89].  Critiquing  is  of  interest  because  it  is  a  way  to  use  knowledge-based 
system  techniques  in  situations  where  autonomous  expert  systems  are  in- 
sqjpropriate.  In  studying  the  approach  we  found  that  in  order  for  a  critic  to  meet 
the  goals  of  cooperative  problem  solving  and  accommodate  user  learning  it  needs 
to  be  able  to  explain  system  knowledge  in  an  individualized  manner.  This  finding 
was  the  primary  motivation  for  choosing  to  investigate  user  modelling  in  this 
thesis.  That  research  required  a  clear  understanding  of  the  paradigm,  so  an  impor¬ 
tant  collaborative  effort  was  to  characterize  critiquing;  this  chapter  is  an  overview 
of  that  effort.  The  term  critiquing  is  intended  to  mean  the  paradigm  or  technique, 
while  we  refer  to  systems  that  use  critiquing  as  “critics”. 

Critics  can  support  users  in  both  problem  solving  and  learning,  they  play 
an  essential  role  in  extending  applications-oriented  design  kits  to  design  environ¬ 
ments,  and  are  an  alternative  to  traditional  expert  system.  The  approach  has  been 
used  successfully  in  diverse  application  domains,  to  both  aid  in  a  cooperative 
problem  solving  process  and  to  provide  support  for  learning.  Ideally  a  single  sys- 
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tern  will  achieve  both  of  these  goals;  such  a  system  will  neeU  the  capability  to 
adapt  explanations  of  its  advice  to  individual  users. 

2.1.  Foundations  for  Critiquing 

Powerful  computer  hardware  makes  it  possible  to  use  computers  in  an 
increasing  range  of  qiplication  areas.  As  technical  complexity  increases,  the  as¬ 
sociated  cognitive  costs  to  master  computers  grow  dramatically  and  limit  our 
ability  to  make  full  use  of  computer  systems.  Systems  that  offer  rich  functionality 
to  their  users  need  to  be  designed  to  be  both  useful  and  usable.  It  is  a  way  to  meet 
the  goals  developed  in  Chapter  1,  providing  support  for  learning  and  for  coopera¬ 
tive  problem  solving.  Critiquing  also  plays  an  important  role  in  the  concept  of 
Design  Environments;  other  work  in  our  group  has  investigated  and  reported  on 
that  line  of  research  [Lemke  89]. 

Cooperative  Problem  Solving.  Critics,  by  their  nature,  operate  in  a 
somewhat  cooperative  manner;  they  can  be  further  enhanced  to  more  fully  achieve 
the  objective  of  having  a  cooperative  problem  solving  system.  They  identify 
proposed  solutions  or  strategies  that  could  be  done  using  an  alternative  approach. 
For  users  to  accept  critics  as  a  useful  feature  of  their  working  environment  they 
need  to  provide  explanations  and,  where  appropriate,  suggest  alternative  solutions. 

Some  shortcomings  of  traditional  expert  systems  were  pointed  out  in 
Chapter  1;  another  one  is  that  these  systems  are  inadequate  when  it  is  difficult  to 
C£q>ture  all  requisite  domain  knowledge.  Because  expert  systems  often  leave  the 
human  out  of  the  process,  they  require  comprehensive  knowledge  that  covers  all 
aspects  of  the  tasks;  all  **  intelligent”  decisions  are  made  by  the  computer.  Some 
domains  are  not  sufficiently  weU  understood,  and  to  create  a  complete  set  of  prin¬ 
ciples  that  capture  them  is  not  possible.  Some  domains  require  considerable  effort 
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in  order  to  acquire  all  relevant  knowledge.  Critics  are  suited  to  these  situations 
because  they  need  not  be  complete  domain  experts.  Critics  can  still  offer  the  user 
helpful  guidance  even  when  their  expertise  is  limited  to  only  some  aspects  of  the 
problem  domain. 

The  traditional  expert  system  approach  is  also  inappropriate  when  the 
problem  is  ill-defined.  This  is  because  the  problem  cannot  be  precisely  specified 
before  a  tentative  solution  is  attempted.  In  contrast,  critics  are  able  to  function 
with  only  a  partial  task  understanding.  Even  when  the  system  contains  only 
general  knowledge  about  the  problem  domain,  it  can  provide  helpful  support  be¬ 
cause  there  exist  general  principles  that  apply. 

Support  for  Learning.  The  computational  power  of  high  functionality 
computer  systems  can  provide  qualitatively  new  learning  environments;  future 
learning  technologies  will  be  multi-faceted  and  support  a  portion  of  the  spectrum 
of  approaches  that  was  shown  in  Figure  1-1.  Some  versions  of  intelligent  tutoring 
systems  developed  in  research  laboratories  allow  the  student  to  exercise  greater 
control  of  the  interaction.  USP  TUTOR  was  reimplemented  in  a  mode  that  per¬ 
mitted  students  to  decide  when  the  system  could  assess  their  work  [Anderson, 
Conrad,  Corbett  89].  Student  performance  on  post  tests  were  equivalent  for  the 
immediate  feedback  (tutor-controlled)  and  the  demand  feedback  (user-controlled) 
versions.  Students  actually  took  longer  to  solve  problems  when  feedback  was  un¬ 
der  their  control  rather  than  the  systems,  however,  the  quality  of  the  learning  ex¬ 
perience  is  not  degraded.  This  has  clear  implications  for  systems  designed  to  sup¬ 
port  learning  in  situations  where  it  is  necessary  for  users  to  provide  the  problem 
specification,  such  as  in  LiSP-CRmc.  As  previously  mentioned,  the  need  to 
provide  knowledge-based  assistance  in  exploratory  learning  environments  is  also 
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recognized.  There  is  a  recognizable  trend  for  designs  of  learning  systems  to  move 
toward  the  middle  of  that  continuum  —  closer  to  the  critiqtiing  paradigm. 

Critics  in  passive  help  systems  may  not  require  users  to  formulate  a 
specific  query,  but  because  they  assist  only  when  called,  they  allow  users  to  retain 
control,  providing  advice  only  when  the  products  or  actions  are  recognized  as  sig¬ 
nificantly  inferior.  By  integrating  working  and  learning,  critics  offer  unique  op¬ 
portunities  for  the  user: 

•  to  understand  purpose  of  or  use  for  the  knowledge  they  are  learning, 

•  to  learn  by  actively  applying  knowledge  rather  than  by  passive  ex¬ 
posure  to  it,  and 

•  to  learn  one  condition  under  which  that  knowledge  can  be  ^rplied. 

A  strength  of  critiquing  is  that  learning  occurs  as  a  natural  byproduct  during  the 
problem  solving  process. 

2.2.  The  Critiquing  Approach 

Human-to-human  critiquing  is  used  in  many  problem  solving  contexts: 
design,  authoring,  student  work  groups,  and  collaborative  research.  People  work¬ 
ing  together  in  these  and  similar  areas  naturally  use  critiquing  as  an  interaction 
style.  Critiquing  is  a  way  to  present  a  reasoned  opinion  about  a  product  or  action 
(see  Figure  2-1),  The  product  could  be  a  computer  program,  a  kitchen  design,  a 
medical  treatment  plan;  an  action  could  be  a  sequence  of  keystrokes  that  corrects  a 
mistake  in  a  word  processor  document  or  a  sequence  of  operating  system  com¬ 
mands.  An  agent  (human  or  machine)  that  is  capable  of  critiquing  in  this  sense 
can  be  called  a  critic.  Critics  can  be  implemented  on  computers  as  a  set  of  rules  or 
specialists  for  the  different  issues  that  may  be  associated  with  a  product;  some¬ 
times  critics  are  the  term  used  for  each  individual  system  component  that  reasons 
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about  a  single  issue.  In  this  project  we  call  the  entire  system  a  critic;  part  of  its 


structure  is  a  composite  rule  set. 
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Figure  2-1;  The  Critiquing  Approach 

This  figure  shows  that  a  critiquing  system  has  two  agents,  a  computer  and  a 
user,  working  in  cooperation.  Both  agents  contribute  what  they  know 
about  the  domain  to  help  solve  some  problem.  The  human’s  primary  role 
is  to  generate  and  modify  solutions,  while  the  computer’s  role  is  to  analyze 
those  solutions,  producing  a  critique  for  the  human  to  ^ply  during  the  next 
iteration  of  this  process. 


Critics  do  not  directly  solve  users’  problems,  but  they  recognize 
deficiencies  in  a  product  and  communicate  those  deficiencies  to  the  users.  Critics 
point  out  errors  and  suboptimal  conditions  that  might  otherwise  remain  un¬ 
detected;  frequently  they  suggest  how  to  improve  the  product.  Users  apply  this 
information  to  fix  the  problems,  seek  additional  advice  or  trigger  requests  for  ex¬ 
planations. 
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It  is  probably  instructive  to  clarify  the  distinction  between  critics  and 
constraints.  A  significant  aspect  of  critiquing  is  that  users  remains  in  control  and 
are  free  to  accept  or  reject  advice  from  the  critic.  Constraints  are  often  “hard 
coded”  into  the  working  environment  of  systems  or  enforced  on  the  user  by  some 
system  process  (e.g.,  a  file  name  extension  in  MS/IX)S  cannot  be  more  than  3 
characters);  they  are  narrowly  focused  criteria  that  must  be  adhered  to  in  order  for 
somediing  to  function  properly.  Critiquing  primarily  focuses  on  improving  the 
ftmctionality  of  a  product  that  is  already  usable.  It  is  possible  to  incorporate  hard 


trigger  that  the  product  must  be  changed  to  comply  with  the  constraint,  or  that  the 
system  already  modified  it  to  comply.  The  majority  of  the  research  on  critiquing 
has  used  critic  expertise  that  is  based  on  what  might  be  called  soft  constraints  or 
design  guidelines. 

Advisors  [Carroll,  McKendree  87]  perform  a  similar  function,  but  they 
are  the  source  of  the  primary  solution.  Users  describe  their  problem,  and  the  com¬ 
puter  advisor  proposes  a  solution.  In  contrast  to  critics,  advisors  do  not  require 
users  to  generate  either  partial  or  complete  solutions  to  the  problem.  Advising  as 
an  interaction  approach  is  best  suited  to  situations  where  one-time  advice  is 
needed.  User  models  are  not  as  significant  in  these  one-shot  affairs,  and  ones  that 
are  used  emphasize  modelling  users’  goals  rather  than  their  domain  expertise.  An 
important  research  issue  is  to  determine  the  commonalities  that  exists  between 
user  models  in  advisory  systems  and  user  models  for  critics. 

To  clarify  any  conflicts  in  terminology,  note  that  the  term  “critic”  was 
also  used  in  the  work  on  planning  systems,  hi  that  context  they  describe  internal 
demons  that  check  for  consistency  during  plan  generation.  For  example,  critics  in 
the  Hacxer  system  [Sussman  75]  discover  errors  in  blocks- world  programs. 


When  the  critics  discover  a  problem,  they  notify  the  planner,  which  modifies  the 
plan  accordingly.  The  NOAH  system  [Sacerdoti  75]  contains  critics  that  recognize 
planning  problems  and  help  to  modify  general  plans  into  more  specie  ones. 
Critics  in  planners  interact  with  the  internal  components  of  the  planning  system; 
critics  in  the  sense  of  this  paper  interact  with  and  critique  the  work  of  human  users. 

23.  The  Critiquing  Process 

The  canonical  process  underlying  critiquing  is  comprised  of  the  sub¬ 
processes  shown  in  Figure  2-2.  Not  all  of  these  processes  are  present  in  every 
critiquing  system;  in  fact,  several  of  these  processes  are  only  conceptual  and 
represent  emerging  research  directions. 

Goal  Acquisition,  Critiquing  a  product  requires  that  the  system  either 
infer  some  limited  understanding  of  the  product’s  intended  purpose  or  be  designed 
to  support  standard  user  goals.  Problem  knowledge  can  be  either  domain 
knowledge  or  goal  knowledge.  If  a  critic  just  has  domain  knowledge  without  im- 
derstanding  the  user’s  goals,  it  can  only  reason  about  characteristics  that  pertain,  in 
a  general  sense,  to  all  products  in  that  domain.  Such  is  the  case  for  LlSP-CRmC;  it 
analyzes  programs  for  syntactic  correctness.  For  a  more  extensive  evaluation  of  a 
product,  an  understanding  of  the  user’s  specific  goals  and  situation  is  desirable.  A 
critic  may  be  able  to  acquire  an  understanding  of  the  user’s  goal  in  several  ways: 

•  Standard  goals  are  built  into  the  system,  in  USP-CRITIC  these  goals 
are  to  produce  code  that  is  either  more  readable  or  more  efficient. 

•  Goals  can  be  recognized  by  observing  users  work  and  the  evolving 
products.  Findings  from  research  on  plan  recognition  in  artificial  in¬ 
telligence  [Schmidt,  Sridharan,  Goodson  78]  would  support  this 
method. 
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Figure  2-2:  The  Critiquing  Process 

Users  initiate  the  critiquing  process  by  presenting  a  product  to  the  critic. 
To  evaluate  the  produa  the  critic  use  a  goal  specification  if  one  is  avail¬ 
able.  To  help  analyze  the  product  some  critics  graierate  a  solution  and 
compare  it  to  the  user’s,  others  analyze  the  user’s  work  directly.  A 
presenter  formulates  a  critique  using  the  product  analysis;  it  provides  ad¬ 
vice  and  explanations.  Critiquing  strategies  and  a  user  modelling  may  be 
used  to  aid  the  presenter.  From  this  output,  the  user  modifies  the  product 
and  the  cycle  can  repeat.  The  essential  processes  and  components  for  a 
system  to  be  considered  a  critic  are  outlined  in  black.  The  objects  in  the 
figure  with  grey  outlines  are  optional  and  in  several  cases  represent 
research  directions. 
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•  A  critic  may  have  access  to  an  explicit  representation  of  the  problem 
to  be  solved,  one  that  encapsulates  a  particular  goal.  A  simple  tech¬ 
nique  is  to  limit  the  possible  goals  and  ask  to  users  to  select  one  from 
that  set. 

Product  Analysis,  The  two  general  approaches  to  critiquing  are  : 
differential  and  analytical.  In  differential  critiquing,  the  system  generates  a  solu¬ 
tion  and  compares  it  to  the  user's  solution.  Analytical  critiquing  checks  the 
product  with  respect  to  predefined  features  and  effects.  They  identify  suboptimal 
features  using  techniques  such  as  pattern  matching  [Fischer  87b],  finite  state 
machines  [Fischer,  Lemke,  Schwab  85],  and  expectation-based  parsers  [Finin  83]. 
Critics  which  use  the  analytical  approaches  do  not  require  a  complete  understand¬ 
ing  of  the  product. 

Critiquing  Strategies.  Critiquing  strategies  and  the  user  model  can  aid 
the  presentation  component  The  critiquing  strategies  determine  what  aspects  of  a 
design  to  critique,  and  when  and  how  to  intermpt  users’  work.  Strategies  differ 
depending  on  whether  the  predominant  use  for  the  system  is  helping  users  to  solve 
problems  or  for  an  educational  application. 

The  manner  in  which  critics  are  integrated  into  a  work  environment 
should  be  chosen  so  that  users  welcome  them  and  find  them  cooperative.  Like 
recommendations  from  colleagues  or  co-workers,  messages  from  a  critic  can  be 
perceived  as  helpful  or  hindering,  and  as  aiding  or  interfering  with  the  accomplish¬ 
ment  of  their  goals.  When  selecting  a  critiquing  strategy  two  factors  to  consider 
are  intrusiveness  and  emotional  impact  on  the  user. 


•  Intrusiveness  is  users’  perception  of  how  much  the  critiquing  is  inter¬ 
fering  with  their  work.  Critics  have  to  trade-off  interfering  too  much 
with  failing  to  provide  sufficient  help.  Factors  to  consider  include 
how  frequently  feedback  occurs,  the  complexity  of  the  tasks,  and  the 
sophistication  of  the  user.  Critics  should  intervene  when  it  is  critical, 
but  interventions  should  not  occur  so  frequently  that  users  are 
bothered  and  become  frustrated. 

•  Emotional  impact  refers  to  how  users  react  toward  the  computer  as  an 
intelligent  assistant.  Computer  critiquing  may  be  more  tolerable  than 
critiquing  from  humans  because  it  can  is  handled  privately  between 
users  and  the  system.  When  dealing  with  a  machine,  users  do  need 
not  to  face  the  negative  aspects  of  shortcomings  in  their  work  being 
exposed  to  other  people  who  might  form  a  negative  opinion. 

The  prime  objective  of  educational  critics  is  to  support  learning;  and  for 
performance  critics,  to  improve  the  product.  Each  type  of  system  has  different 
requirements  for  selecting  jqjpropriate  strategies.  A  performance  critic  should  help 
users  create  high-quality  products  in  the  least  amount  of  time  while  conserving 
resources.  Learning,  although  not  the  primary  concern  of  performance  systems, 
occurs  as  a  by-product  of  the  user  and  critic  interaction.  Educational  critics  try  to 
maximize  the  information  or  skills  that  users  acquire  and  retain  for  future  use. 
Most  performance  critics  evaluate  the  product  as  a  whole  and  determine  if  it  can 
be  changed  to  achieve  a  higher  quality  result.  Some  critics  selectively  critique 
based  on  a  policy  specified  by  the  user.  Educational  critics  need  more  complex 
intervention  strategies  to  maximize  information  retention  and  users’  motivation. 
For  example,  an  educational  critic  may  forego  the  opportunity  to  critique  if  it  oc¬ 
curs  too  soon  after  a  previous  critiquing  episode.  Continuous  critiquing  without 
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giving  users  a  chance  to  explore  their  own  ideas  can  become  intrusive  and  impact 
motivation. 

Existing  critics  operate  primarily  in  the  negative  mode  by  pointing  out 
suboptimal  aspects  of  the  user’s  product  or  solution.  A  positive  critic  should 
recognize  and  point  what  is  good  about  a  user’s  solution.  For  performance  critics, 
a  positive  approach  can  help  users  recognize  the  good  aspects  of  their  work.  For 
educational  critics,  positive  critiquing  can  reinforce  desired  behavior  and  thereby 
aid  learning. 

Intervention  strategies  determine  when  and  how  a  critic  intercedes. 
Active  critics  control  intervention  because  they  can  critique  a  product  or  action  at  a 
time  of  their  choosing.  They  are  active  agents  continuously  monitoring  user  ac¬ 
tions.  Passive  critics  are  explicitly  invoked  whenever  users  want  an  evaluation. 
Most  passive  critics  are  able  to  evaluate  partial  products  but  not  individual  user 
actions. 

Adaptation  Capability.  To  avoid  repeating  the  same  type  of  advice  and 
to  accommodate  different  users  with  different  preferences  and  skills,  a  critiquing 
system  needs  an  adaptation  capability.  A  critic  that  persistently  critiques  users 
using  positions  with  which  they  disagree  is  unacceptable,  especially  when  the 
critique  is  intrusive.  A  critic  that  constantly  repeats  an  explanation  that  the  user 
already  knows  is,  similarly,  unacceptable. 

There  are  two  aspects  to  an  adaptation  c^ability:  critics  can  be  ad£q>t- 
able  or  adaptive.  Systems  are  adt^table  if  a  user  can  change  their  behavior  or 
knowledge:  more  recent  research  has  called  these  systems  “end  user  modifiable’’ 
[Fischer,  Girgensohn  90].  On  the  other  hand,  an  adaptive  system  is  one  that 
automatically  changes  its  behavior  based  on  observed  or  inferred  information.  An 
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adaptation  capability  can  be  implemented  by  disabling  or  enabling  the  firing  of 
particular  critic  rules,  by  allowing  the  user  to  modify  or  add  rules,  or  by  making 
the  critiquing  strategy  depend  on  an  explicit,  dynamic  individual  user  model. 

Explanation.  Explanations  are  desirable  and  necessary  in  most 
knowledge-based  systems  [Swartout  81;  Teach,  Shortliffe  84].  Critics  need  to  be 
able  to  explain  their  rationale  so  users  can  assess  the  critique  and  decide  how  to 
deal  with  the  advice.  Knowing  why  a  product  was  critiqued  helps  users  to  learn 
underlying  principles  and  avoid  similar  problems  in  the  future.  In  a  critiquing  sys¬ 
tem,  explanations  can  focus  on  the  specific  differences  between  the  system’s  and 
the  user’s  solutions,  the  rationale  underlying  the  critique,  or  on  violations  of 
general  guidelines. 

Advisory  Capability.  Critics  detect  suboptimal  aspects  of  a  user’s 
work;  this  is  the  triggering  condition  for  a  critiquing  episode.  When  an  episode 
stops  here,  the  user  is  required  to  generate  and  inqjlement  any  changes  to  the  cur¬ 
rent  product.  One  inprovement  on  the  process  is  for  the  critic  to  suggest  alter¬ 
natives;  these  we  call  solution- generating  critics.  Another  is  to  provide  the  critic 
the  ability  to  explain  or  direct  users  toward  information  that  increases  their  under¬ 
standing.  User  models  play  a  role  in  facilitating  this  part  of  extending  the 
paradigm;  it  is  a  subject  that  will  be  discussed  in  detail  in  the  remainder  of  this 
thesis. 

2.4.  Survey  of  Critiquing  Systems 

This  section  provides  an  overview  of  critiquing  systems  that  play  an  im¬ 
portant  role  in  the  development  of  the  paradigm  or  illustrate  an  interesting  a^ct 
of  the  theory.  In  addition  to  LlSP-CRmc,  the  Human  Computer  Communications 
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Group  has  developed  JANUS  [Fischer,  McCall,  Morch  89a]  and  FRAMER  [Lemke 
90]  to  deepen  our  understanding  of  the  critiquing  paradigm.  LlSP-CRrnc  will  be 
discussed  in  detail  in  Chapter  3.  Not  all  systems  developed  by  other  researchers 
are  described  by  their  authors  using  the  terminology  presented  here,  but  they  do  fit 
into  the  critiquing  framework.  Because  the  range  of  systems  covers  diverse  sp- 
plication  domains,  a  claim  can  be  made  that  critiquing  has  general  application  as  a 
central  approach  to  building  knowledge-based  systems.  During  the  process  of 
developing  the  user  modelling  framework  an  attempt  was  made  to  retain  this 
perspective  of  domain  generality. 

Critiquing  is  attractive  because  of  its  generality  across  a  wide  range  of 
domains,  such  as  medicine;  electronic  circuit  design;  and  support  for  education, 
writing,  programming,  and  text  editing.  This  section  briefly  surveys  the  critiquing 
systems  in  these  domains  that  were  studied.  Most  of  the  systems  discussed  here 
were  developed  as  research  vehicles,  but  a  few  are  successful  commercial  applica¬ 
tions. 

The  West  system  pioneered  many  of  the  fundamental  ideas  behind  the 
critiquing  paradigm.  It  was  an  early  effort  to  build  a  computer  coach  [Burton, 
Brown  82]  that  teaches  arithmetic  skill  in  a  gaming  environment  (a  game  called 
‘‘How  the  West  was  won”).  The  goal  was  to  augment  an  informal  learning  ac¬ 
tivity  with  a  computer  coach,  retaining  the  engagement  and  excitement  of  a  stu¬ 
dent  directed  activity  while  providing  context-sensitive  advice  on  how  students 
can  improve. 

Several  important  ideas  were  pioneered  in  WEST.  It  builds  a  bridge  be¬ 
tween  open  learning  environments  and  tutoring  in  order  to  support  what  is  called 
guided  discovery  learning.  A  model  of  each  user  prevents  the  coach  from  being 
too  intrusive.  The  system  uses  diagnostic  modeling  strategies  to  infer  problems 
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from  students’  actions.  WEST  detennines  the  causes  of  suboptimal  behavior  by 
comparing  the  solution  of  a  built-in  expert  with  the  student’s  solution.  In  this 
manner,  the  student  model  is  acquired  by  a  process  called  differential  student 
modelling.  The  system  infers  models  of  students  in  terms  of  the  “issues”  on 
which  they  are  weak  (mathematical  procedures  and  game  playing  strategies).  In¬ 
tervention  and  tutoring  strategies  are  explicitly  represented  in  the  system  and  make 
use  of  information  contained  in  the  model  to  enable  the  coach  “to  say  the  right 
thing  at  the  right  time”  and  provide  coherence  to  that  feedback. 

Medical  applications.  Several  researchers  in  the  domain  of  medicine 
have  embraced  the  critiquing  approach.  In  general,  these  systems  aid  medical  per¬ 
sonnel  in  patient  diagnosis  and  treatment.  Clancey  first  proposed  a  critiquing  ap- 
proach  to  user-system  interaction  for  expert  medical  consultation  systems 
[Qancey  84].  Miller  and  colleagues  at  Yale  Medical  School  did  the  most  im¬ 
plementation  work  in  this  area,  developing  systems  which  assist  medical  persoiuiel 
by  analyzing  plans  for  the  prescription  of  medication,  managing  its  administration, 
monitoring  the  use  of  a  ventilator,  and  administration  of  anesdietics  [Miller  86]. 

The  most  extensively  developed  system  is  ATTENDING  [Miller  86].  It 
uses  the  differential  critiquing  sq^proach,  parsing  the  physician’s  plan  starting  with 
the  top-level  decisions  and  at  each  step  trying  to  find  alternatives  that  have  lower 
or  equal  patient  risks.  The  system  works  from  the  physician’s  solution  to  a 
system’s  solution  to  insure  that  it  is  as  close  to  the  physician’s  as  possible,  this 
makes  the  critique  more  helpful  and  easier  to  understand. 

Differential  critiquing  is  also  used  in  one  version  of  ONCOCIN,  an  expert 
system  for  cancer  therapy  [Langlotz,  Shortliffe  83].  The  developers’  goal  was  to 
eliminate  the  need  to  override  the  system  when  justifying  minor  deviations  from 
the  dierapy  plan  for  the  convenience  of  the  patient. 
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Roundsman  [Rennels  87;  Rennels,  Sbortiliffe,  Stockdale,  Miller  89]  is 
a  critic  in  the  domain  of  breast  cancer  treatment  diat  bases  its  critique  on  studies 
from  the  medical  literature.  It  is  a  passive  critic  with  explicit  goal  specification. 
Text  in  the  literature  database  serves  as  a  domain  knowledge  set  that  is  not  inter* 
pretable  by  the  system,  but  stored  in  a  “canned”  form;  associated  with  each  case 
description  in  the  database  are  a  set  of  case-factors  that  can  be  used  for  retrieval. 
ROUNDSMAN  can  automatically  provide  the  case  descriptions  as  a  form  of  detailed 
explanation.  Redundancy  is  a  problem  and  no  facilities  are  available  for  users  to 
followup  on  the  advice  or  textual  descriptions  found  in  the  literature.  The  system 
is  successful  because  there  is  a  close  mapping  between  the  ctirrent  case  charac¬ 
teristics  (e.g.,  tumor  size,  location,  (patient  age,  etc)  and  recorded  medical  case 
studies.  It  could  be  viewed  as  critiquing  system  that  uses  case-based  reasoning, 
except  that  the  system  does  not  really  attempt  to  understand  die  cases,  rather  it 
knows  how  to  match  the  symptoms  of  the  patient  imdergomg  diagnosis  with  those 
cases. 

Circuit  design.  Several  research  and  commercial  systems  use  a  critiqu¬ 
ing  approach  for  enhancing  digital  circuit  designs.  (FRITTER  [Kelly  85]  is  a  design 
aid  for  digital  circuits.  It  uses  a  schematic  diagram  and  a  set  of  specifications  to 
evaluate  the  circuit  using  axudysis  techniques  and  knowledge  about  primitive  com¬ 
ponents.  The  evaluation  report  includes  information  about  how  well  the  circuit 
will  work. 

A  commercial  system  developed  at  NCR  is  the  Design  Advisor™ 
[Steele  88].  It  is  an  expert  system  that  provides  advice  on  application-specific 
integrated  circuit  designs.  The  Design  Advisor  arudyzes  the  performance,  tes¬ 
tability,  manufacturability,  and  overall  quality  of  CMOS  semi-custom  VLSI 
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designs.  Its  knowledge  is  a  hierarchy  of  design  attributes  compiled  from  a  study 
of  major  problems  in  commercial  VLSI  designs.  Critiquing  is  not  interactive  but 
done  using  a  batch  mode;  designers  submit  proposed  circuits  and  the  system 
returns  the  analysis  to  them  for  any  actual  design  modifications. 

Discovery  learning.  A  suite  of  three  computer-based  coaching  systems 
for  discovery  learning,  developed  at  LRDC,  University  of  Pittsburgh,  are  based  on 
critics.  These  systems  each  address  a  different  domain:  SMITHTOWN  — 
microeconomics  [Raghaven,  Schultz,  Glaser,  Schauble  90],  VOLTAVILLE  —  direct 
current  electricity  [Glaser,  Raghaven,  Schauble  88],  and  REFRACT  —  geometrical 
optics  [Riemann,  Raghaven,  Glaser  88].  These  discovery  environments  are 
designed  to  build  scientific  inquiry  skills.  Active  critics  judge  the  efficiency  of  the 
processes  used  to  build  scientific  theory  and  inform  users  about  errors  that  charac¬ 
teristically  trap  less  successful  students  as  well  as  guide  them  to  effective 
strategies. 


Decision  making.  The  DecisionLab  system  developed  at  the  European 
Computer  Industry  Research  Center  [Schiff,  Kandler  88]  ^>plies  the  critiquing  £q>- 
proach  to  guide  users  in  managerial  decision-making.  DecisionLab  provides  con¬ 
structive  feedback  on  a  user’s  management  plan  in  a  simulation  game.  The  user 
gets  critiqued  whenever  they  attempt  a  non-optimal  approach.  This  system  in¬ 
tegrates  a  critic  and  a  simulation  exercise. 

Mili  is  investigating  how  to  apply  the  critiquing  approach  to  improve  the 
performance  of  decision  makers  in  the  context  of  their  actual  work  with  a  system 
called  DECAD.  It  has  not  actually  been  built,  but  is  designed  to  watch  over  the 
shoulder  of  the  decision  maker,  inteijecting  advice  or  a  critique  when  appropriate 
[Mili  88].  In  the  area  of  research  into  decision  support  systems,  investigators 
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place  critiquing  into  a  class  of  knowledge-based  systems  called  "'active  and  sym¬ 
biotic  decision  support  systems”  [MUi,  Manheim  88]. 

An  operational  symbiotic  decision  support  system  to  support  steel  mill 
operations  is  being  developed  by  Manheim  and  colleagues  [Manheim,  Srivastava, 
Vlahos,  Hsu,  Jones  90].  A  manager  develops  a  plan  using  a  commercially  avail¬ 
able  production  planning  and  scheduling  system  which  includes  a  mathematical 
model,  heuristic,  and  optimization  techniques,  that  plan  is  compared  to  a  system 
developed  plan  using  differential  critiquing. 

Curriculum  development  The  Alberta  Research  Council  (Canada) 
and  a  company  called  Computer  Based  Training  Systems  developed  and  are 
maiketing  a  knowledge-based  system  which  provides  assistance  with  curriculum 
and  course  development  [Wi^nd,  Jones  88].  An  expert  module  monitors  cur¬ 
riculum  and  course  development,  intervening  when  necessary  or  when  assistance 
is  requested.  The  expert  monitor  can  suggest  what  to  do  next,  where  to  firul  ex¬ 
amples  or  how  to  get  more  help. 

Authoring.  Critiquing  systems  have  been  developed  diat  help  writers 
make  their  text  more  readable  or  help  writers  learn  more  efficient  text  editing 
strategies  with  which  to  produce  that  text.  WANDAH  [Friedman  87]  is  a  system 
that  assists  authors  in  all  phases  of  writing;  it  is  cotiunercially  available  for  per¬ 
sonal  computers  as  HB  J  Writer™.  Text  which  need  not  be  a  corr^leted  document 
can  be  subjected  to  one  of  four  sets  of  reviewing  and  revising  aids  that  go  over  the 
written  work;  the  system  provides  feedback  on  stractural  problems,  and  recom¬ 
mends  revisions. 

ACTIVIST  is  an  active  help  system  for  a  screen-oriented  text  editor  that 
monitors  users’  activities.  It  recognizes  sequences  of  actions  that  are  intended  to 
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achieve  one  of  the  twenty  different  goals  known  to  the  system;  some  examples  are 
deleting  a  word  or  moving  the  cursor  to  the  end  of  the  current  line.  ACTIVIST 
critiques  the  user  after  three  suboptimal  executions  of  a  task  type.  After  a  certain 
number  of  correct  executions,  the  system  will  no  longer  watch  for  that  plan.  It 
ceases  to  critique  actions  when  a  user  ignores  its  suggestions  for  those  actions. 
This  system  integrates  a  user  model;  that  model  plays  a  central  role  in  informing 
the  system  when  to  intervene,  when  to  discontinue  looking  for  a  plan,  or  when  to 
ignore  user  actions.  The  user  model  represents  plans  or  strategies  that  users  may 
be  following,  ones  can  they  execute  optimally,  and  others  they  prefer  not  to 
change. 

Software  development.  PROLOG  explaining  [Coombs,  Alty  84]  is 
designed  to  enhance  a  programmer’s  understanding  of  PROLOG,  thereby  helping 
the  user  to  develop  a  better  understanding  of  the  language.  Users  are  shown  some 
PROLOG  code  and  asked  to  construct  an  explanation  of  that  code;  the  system 
critiques  that  explanation. 

The  GRACE  system  developed:  at  the  NYNEX  Artificial  Intelligence 
Laboratory  is  a  multi-faceted  learning  environment  for  COBOL  programming  diat 
integrates  a  critic,  a  tutor,  and  a  hypertext  information  base.  When  the  system  is 
functioning  as  a  critic,  it  can  adopt  a  tutoring  mode  to  give  remedial  problems;  and 
conversely,  when  functioning  as  a  tutor  the  student  can  decide  to  explore  in  the 
critiquing  mode.  The  tutor  is  a  production  rule-based  system  modelled  after  the 
LISP  TUTOR  [Anderson,  Reiser  85].  The  tutor  portion  of  the  system  contains  a  stu¬ 
dent  model  that  is  an  overlay  of  the  productions  contained  in  the  system.  That 
model  is  not  shared  with  the  critic,  nor  does  the  critic  attempt  to  tailor  its  inter¬ 
action  to  the  individual. 
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Kate  [Fickas,  Nagarajan  88]  critiques  software  specifications  (for 
automated  library  systems)  that  are  represented  in  an  extended  Petri  net  notation. 
Its  knowledge  is  represented  as  “cases’*  consisting  oft  a  pattern  describing  a  be¬ 
havior  in  a  specification,  links  to  one  or  more  goals,  simulation  scenarios,  and 
canned  text  descriptions.  The  critic  evaluates  the  specification  with  respect  to 
goals  or  policy  values  given  by  the  user. 

Mechanical  design.  Feedback  Mini-Lab  [Forbus  84]  was  built  as  a 
follow-on  to  the  original  woric  on  the  STEAMER  system.  It  is  an  environment  in 
which  simulated  devices,  such  as  steam  plant  controllers,  can  be  assembled  and 
operated.  Students  can  assemble  a  device  from  the  building  blocks.  Feedback 
Mini-Lab  is  designed  to  facilitate  student  understanding  of  control  components. 
Mini-lab  generates  code  specifications  to  produce  the  simulation  for  the  device. 
After  constructing  their  device,  students  can  ask  the  system  for  a  critique. 

2.5.  Limitations  of  Current  Critics  and  Future  Research  Issues. 

One  features  that  is  a  strength  of  the  critiquing  approach  is  also  a  poten¬ 
tial  weakness.  Supporting  users  in  their  own  doing  means  that  detailed  assump¬ 
tions  about  what  a  user  might  do  cannot  be  built  into  the  system.  Our  systems 
have  a  limited  understanding  of  users’  goals.  This  restricts  the  amount  of  assis¬ 
tance  and  goal-oriented  analysis  that  critics  can  provide  in  comparison  to  systems 
such  as  PROUST  [Johnson,  Soloway  84],  which  have  a  deep  understanding  of  a 
limited  set  of  problems. 

Most  rule-based  critics  do  not  have  an  explicit  representation  of  all  the 
rationale  for  their  knowledge.  Therefore,  to  capture  enough  domain  knowledge  to 
provide  explanations,  these  systems  need  more  abstract  representations  of  the  q)- 
plication  domain. 
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Critics  should  ideally  have  inspectable  knowledge  structures  so  that 
users  can  modify  and  augment  them.  This  does  not  mean  that  users  will  have  to 
possess  detailed  programming  knowledge.  As  a  minimum  users  should  be  able  to 
deactivate  (and  reactivate)  individual  rules  according  to  their  needs  and  goals. 
With  sufficient  inference  and  user  modeling  capabilities,  systems  might  be  able  to 
do  dynamic  adaptation. 

Currently,  most  critics  support  only  a  “one-shot  dialog”  [Aaronson, 
CarroU  87].  They  respond  to  actions  taken  by  the  user,  in  some  cases  they  give 
suggestions  and  explanations  but  none  have  the  ability  to  ad2q)t  those  explanations 
to  an  individual  user.  Human  critiquing  is  a  more  cooperative  activity,  during 
which  an  increased  understanding  of  the  problem  develops.  Research  on  how  to 
incorporate  more  of  the  characteristics  of  human-to-human  collaborative  effort 
into  these  systems  is  needed.  This  hqrpens  to  be  one  of  three  directions  for 
research  suggested  for  overcoming  the  limitations  of  human-machine  interaction 
that  were  suggested  in  the  analysis  of  situated  action  [Suchman  87]. 

2.6.  Summary  ^ 

Critiquing  can  be  used  as  an  approach  to  designing  knowledge-based 
computer  systems  that  support  human  work  and  learning.  Critics  are  important 
steps  towards  the  creation  of  more  useful  as  well  as  more  usable  computer  sys¬ 
tems.  Some  of  these  systems  will  have  elaborate  problem  understanding;  more 
commonly,  they  will  have  limited  yet  helpful  c^abilities;  such  as  modelling  their 
individual  users.  Research  on  user  modelling  in  other  paradigms,  such  as  tutoring 
and  advisory  systems,  can  establish  ideas  and  techniques  that  might  be  of  use  in 
critics.  A  review  of  that  user  modelling  research  and  theory  will  be  the  subject  of 
the  next  chapter. 
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USER  MODELLING 

This  chapter  examines  related  research  in  user  modelling  and  describes  a 
general  framework  for  the  user  modelling  component  of  a  system.  Two  related 
research  areas  have  attempted  to  integrate  idiosyncratic  models  of  users.  Research 
in  Intelligent  Computer  Aided  Instruction  (ICAI)  systems,  most  often  referred  to 
as  Intelligent  Tutoring  Systems  (or  ITS),  use  models  of  their  studei^ts  to  guide  the 
instmctional  interaction  [VanLehn  88].  Artificial  intelligence  techniques  are  the 
basis  for  modelling  users  of  advice  giving  dialog  systems  [Kobsa,  Wahlster  89]. 
In  the  work  on  user  models  in  these  areas  I  found  some  concepts  that  provide 
foundations  for  a  user  modelling  framework  to  support  cooperative  problem  solv¬ 
ing,  and  some  specific  ideas  that  were  ad^ted  for  a  user  modelling  in  critiquing  . 
Those  foundations  will  be  discussed  and  the  conceptual  architecture  for  a  user 
modelling  component  presented. 

In  the  wider  context  of  human-computer  interaction  the  term  “user 
model”  is  over-used;  it  has  been  applied  to  mean  three  different  models: 

1.  the  conceptual  model  a  user  forms  of  a  system  (more  precisely  a 
user’s  model  [Norman  86]), 

2.  a  models  that  represents  the  typical  users  of  a  system  as  a  class  and 
are  used  to  aid  in  designing  systems,  and 

3.  models  of  a  specific  user  inferred  by  the  system,  such  as  the  ones 


investigated  in  this  research. 
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Models  in  the  first  sense  are  conceptual  models  that  provide  part  of  a  foundation 
for  understanding  the  process  of  human-computer  interaction.  The  second  class  of 
models  above  are  psychological  models  developed  by  and  for  the  analysis  of 
human  behavior  when  interacting  with  computers.  They  play  an  important  role  in 
guiding  system  development  and  research  in  the  psychology  of  human-computer 
interaction;  important  examples  are  the  GOMS  model  [Card,  Moran,  Newell 
83]  and  cognitive  complexi^  theory  [Kieras,  Poison  85].  Research  in  this  area 
also  compares  these  models  to  one  another  for  given  tasks  [Moran  81;  Young,  Bar¬ 
nard,  Simon,  Whittington  89].  In  the  future  there  is  the  possibility  that  these  two 
lines  of  research  will  converge  to  the  point  where  psychological  models  can  also 
serve  as  a  basis  for  idiosyncratic  representations  of  the  individuals  using  a  system, 
but  neither  research  area  has  matured  to  a  point  where  that  is  presently  feasible. 
The  distinctions  are  clarified  here*  to  insure^  there  is  no  confusion  concerning  the 
interest  of  this  research  —  it  is  user  models  in  the  sense  of  the  third  caiegoiy. 

An  argument  has  been  put  forth  that  the  lack  of  commercial  systems 
with  user  modelling  is  evidence  for  a  failure  in  the  research,  perhaps  an  argument 
for  discontinuing  it  altogether  [Williams  90].  My  position  is  that  this  view  is  en¬ 
tirely  too  pessimistic  and  that  the  reasons  we  do  not  yet  find  the  technology  in 
general  use  are  predominantly  organizational  and  ecmiomic.  Specifically,  there 
are  four  possible  explanations.  First,  the  technology  is  not  fuUy  mature  and  ad¬ 
ditional  research  is  needed,  ergo  the  argument  for  purstiing  this  line  of  research. 
Second,  the  paradigms  for  the  associated  systems  that  use  such  models  are  neither 
completely  understood  themselves,  nor  fully  developed  to  the  point  of  being  com¬ 
mercially  viable  —  for  ICAI  that  means  computer-based  instractional  methodol¬ 
ogy,  and  for  advisory  dialog  systems  the  ability  to  adequately  generate  natural  lan¬ 
guage.  Third,  the  computational  environments  that  run  these  systems  (most  AI 
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applications  for  that  matter)  are  expensive  and  scaling  die  techniques  to  Ht  them  to 
more  common  platfoims  is  a  significant  area  for  research  in  itself.  Fourth,  the 
techniques  are  not  generally  understood  by  designers  and  builders  of  software  to 
support  commercial  ^qiplications,  a  not  uncommon  phenomena  in  area  of  computer 
science  and  the  reason  we  find  suboptimal  system  design  approaches  in  everything 
from  text  editors  to  commercial  databases  in  the  marketplace.  Hie  complete  story 
is  likely  some  combination  of  these  reasons,  and  arguments  based  on  personal  con¬ 
jectures  of  what  win  be  successful  backed  primarily  widi  observations  about  cur¬ 
rent  commercial  computational  systems,  should  not  dissuade  us  from  pursuing  ad¬ 
ditional  understanding  and  new  approaches  to  solving  any  problems  in  computer 
science,  to  include  user  modelling. 

3.1.  An  Overview  of  User  Modelling  Research 

One  survey  of  user  modelling  definitions  together  with  an  effort  to 
correlate  that  research  in  both  human-computer  interaction  and. intelligent  tutoring 
resulted  in  a  useful  taxonomy  based  on  who  owns  the  model  and  its  function 
[Murray  88].  It  was  still  necessary  to  conduct  my^  own.  study;  There  was  a  need 
to  understand  other  research  at  level  of  their  implementation  methodologies  in  or¬ 
der  that  I  could  determine  how  the  models  woik,  and  then  decide  if  the  techniques 
used  have  could  be  used  in  the  user  modelling  component  which  we  wanted  to 
build.  The  most  widely  reported  examples  of  woddng  techniques  for  user  modell¬ 
ing  are  those  developed  to  support  intelligent  tutoring  and  dialog  advisory  systems. 
This  section  describes  the  analysis  of  those  areas;  awareness  of  that  research 
provides  both  implementation  ideas  and  has  helped  to  determine  the  several  re¬ 
quirements  for  a  user  model  able  to  support  cooperative  problem  solving. 


3.1.1.  Student  Models  in  Intelligent  CAI 
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User  Models  in  intelligent  tutoring  systems,  called  student  models,  have 
been  the  subject  of  ongoing  research  for  about  a  decade,  there  is  significant  litera¬ 
ture  surveying  and  discussing  that  work  [Sleeman,  Brown  82;  Wenger  87;  Poison, 
Richardson  88;  Psotka,  Massey,  Mutter  88b;  VanLehn  88].  Student  models  are 
derived  from  knowledge  in  the  system  such  as  rules,  concepts,  or  strategies  for 
learning  a  skill.  The  user’s  knowledge  state  is  represented  as  a  perturbation  of  that 
domain  model  —  popular  approaches  are  overiays  to  represent  the  portions  of  the 
knowledge  base  that  a  student  knows,  and  a  bug  models  that  represent  user  mis¬ 
conceptions  about  the  domain.  Some  ITS  student  models  combine  these  two  tech¬ 
niques  into  a  comprehensive  representation.  Differential  modelling  is  the  term  of¬ 
ten  used  for  these  techniques  [Wilkins,  Clancey,  Buchanan  88].  Several  systems 
which  are  frequently  cited  as  using  successful  approaches  were  studied  in  detail. 

The  WEST  projea  was  previously  discussed  in  Chapter  2.  It  pioneered 
the  differential  modelling  approach  [Wenger  87].  Student  behavior  is  modelled  in 
terms  of  the  issues  they  understand  and  correctly  apply.  Their  behavior  is  com¬ 
pared  to  an  expert’s  under  the  same  conditions  to  determine  their  mastery  of  par¬ 
ticular  issues.  The  system  finds  an  issue  a  student  does  not  know,  then  selects  an 
abstract  explanation  for  that  issue  from  prestored  text.  There  are  limitations  to  this 
approach,  when  compared  to  the  conditions  under  which  critiquing  systems  must 
function.  The  domain  has  a  number  of  properties  that  are  not  characteristic  of  the 
domains  in  which  critics  are  needed.  The  computer  expert  is  able  to  play  an  op¬ 
timal  game  because  there  is  a  best  solution,  and  it  can  interpret  all  alternative  stu¬ 
dent  actions.  In  WEST  it  is  possible  to  identify  students’  bugs,  whereas  in  other 
domains  one  can  only  speak  of  “suboptimal”  behavior.  The  set  of  issues,  on 
which  the  methodology  is  based,  is  closed  for  the  game.  How  the  West  Was  Won, 
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while  it  is  frequently  open-ended  in  other  domains.  The  user’s  task  goal  is  ob¬ 
vious;  it  is  to  win  the  game  while  obeying  its  rules,  another  simplifying  assump¬ 
tion  which  does  not  apply  to  many  other  domains.  The  explanation  strategy  in 
West  presumes  that  the  advice  given  is  self-explanatory  because  it  contains  a 
good  illustrating  example.  Two  ideas  developed  in  WEST  are  of  use  in  this 
research.  One  is  the  idea  that  students’  actions  in  the  ongoing  dialog  with  the 
system  contain  information  that  can  be  used  to  analyze  the  state  of  their 
knowledge.  Another  is  the  notion  that  knowing  this  state  provides  a  mechanism 
for  guiding  presentation  of  new  knowledge  by  the  computer  coach. 

The  genetic  graphs  i^roach  was  first  developed  for  the  WUSOR-H  com¬ 
puter  coach  as  a  way  to  overlay  domain  knowledge  with  a  learner-oriented  linkage 
of  rules  [Goldstein  82].  The  rules  are  represented  as  nodes  in  a  graph  model.  The 
domain  for  the  WURSOR  systems  (three  versiorxs  were  developed  in  all)  is  an  ex¬ 
ploration  adventure  computer  game  called  WUMPUS.  In  another  project,  the 
genetic  graph  s^roach  was  used  as  a  basis  for  modelling  procedural  skills  in  two 
quite  different  domains,  one  mental,  subtraction,  and  the  other  motor,  ballet 
[Brech,  Jones  88].  That  research  validated  the  generality  of  the  approach  and  en¬ 
hanced  general  understanding  of  the  paradigm.  The  nodes  in  the  genetic  graph 
represent  domain  entities,  such  as  skills,  facts,  rules,  or  concepts,  all  elements  of 
expertise.  The  links  between  nodes  capture  the  processes  by  which  a  student  can 
learn  those  domain  entities.  A  system  component  known  as  a  psychologist  inter¬ 
rogates  the  user  model  to  determine  what  to  teach  next;  it  is  also  the  entrusted  with 
maintaining  that  model.  Processes  represented  in  the  links,  such  as  generalization 
or  analogy,  indicate  methods  by  which  students  can  learn  a  new  piece  of 
knowledge  starting  from  one  they  have  mastered.  The  system  can  determines  a 
pedagogical  qiproach  because  the  student  model  is  an  overlay  of  the  graph  with 
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naadced  nodes  representing  skills  or  knowledge  that  students  possess.  The  links 
between  the  nodes  provide  paths  to  die  target  knowledge;  they  represent  possible 
strategies  for  ‘‘teaching”  that  knowledge. 

Genetic  gr^hs  are  normative  models  that  define  in  their  link  structure 
the  manner  in  which  knowledge  in  a  specific  domain  can  be  acquired  by  a  student; 
this  is  an  inherent  limitation.  The  gr^hs  have  to  explicitly  capture  in  the 
representation  all  possible  ways  for  a  user  to  learn  a  domain  entity,  requiring  sig¬ 
nificant  up-front  analysis.  To  construct  the  graph  a  system  designer  has  to  deter¬ 
mine  the  domain  entities  and,  for  each  one,  all  methods  by  which  a  student  could 
learn  one  entity  when  they  already  know  another.  This  restriction  is  similar  that  of 
to  traditional  Computer-Aided  Instruction  which  has  to  algorithmically  pre-specify 
the  possible  paths  through  course  material.  Genetic  graphs  permit  more  flexibility 
in  that  users  can  traverse  the  graph  during  learning  according  to  an  arbitrary,  radier 
than  predetermined  path,  but  the  path  must  be  one  that  has  been  c^tuied  and 
represented  in  the  graph. 

Qancey  compiled  survey  of  student  models  in  ‘‘Al-based  instructional 
programs”  [Qancey  86]  that  contained  a  useful  framework  for  research.  He 
characterizes  student  me  is  as  qualitative  models  in  the  sense  that  they  predict 
how  the  modelled  learner  will  solve  selected  problems,  as  opposed  to  representing 
the  student  with  numeric  measures  of  achievement.  The  system  runs  the  model  as 
a  simulation  of  that  student  to  predict  and  explain  behavior.  Inconsistencies  be¬ 
tween  the  prediction  and  actual  student  activities  serve  as  a  source  of  diagnosis  to 
improve  the  student  model;  in  some  cases  capturing  new  knowledge  (new  to  the 
student  model,  that  is)  that  the  student  possesses,  in  others  identifying  misconcep¬ 
tions  or  bugs,  and  in  still  others  doing  both.  User  models  for  cooperative  problem 
solving  systems  will  not  (and  caimot)  be  predictive  because  of  the  complexity  of 
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the  domains  and  because  the  open-ended  problem  solving  situations  in  which  they 
operate,  preclude  the  system  from  being  able  to  generate  a  con^lete  problem  solu¬ 
tion.  If  the  system  is  not  able  to  solve  any  problem  in  the  domain,  then  it  follows 
that  it  will  not  be  possible  to  use  such  an  approach  in  juxtaposition  with  a  qualita¬ 
tive  user  model  to  predict  user  actions.  One  aspect  of  this  study  that  fits  with  our 
analysis  of  what  is  required  for  modelling  users  of  cooperative  knowledge-based 
systems  is  Clancey’s  finding  that  existing  instructional  programs  had  to  be  en¬ 
hanced  by  second-generation  knowledge  representation  technique.  As  will  be  dis¬ 
cussed  in  Chapter  5  a  similar  requirement  for  enhancing  critiquing  systems  to 
more  fully  support  cooperative  problem  solving  and  learning  precipitated  the 
development  of  a  conceptual  model  for  the  domain  of  LISP. 

The  idea  of  student  models  based  on  the  misconceptions  or  bugs  (also 
called  mal-rules  in  some  research)  that  students  holds  about  die  domain  was  a 
theme  in  several  ICAI  research  projects  besides  WEST  [Brown,  VanLehn  80;  Van- 
Lehn  88].  Those  results  did  not  play  a  role  in  this  woik  because  the  needs  of  our 
systems  emphasize  representing  and  using  what  users  know  about  the  domain 
rather  than  correcting  deficiencies  in  that  knowledge. 

It  is  is  significant  that  ICAI  systems  are  able  to  solve  any  problem  on 
which  their  users  (the  students)  will  work.  Within  their  application  domain  they 
will  restrict  students  to  those  problems.  This  allows  them  to  use  more  detailed  and 
specific  model  inferencing  techniques  than  those  available  to  systems  serving  in 
more  open-ended  problem  solving  situations.  The  requirement  for  our  systems  to 
have  generality  means  that  the  techniques  that  are  often  used  in  student  modelling 
are  often  not  robust  or  general  enough  to  support  real  world  problem  solving.  The 
problem-space  limitations  that  are  imposed  by  tutoring  systems  are  what  make 
them  effective  at  teaching  within  those  restrictions,  and  also  what  enables  them  to 
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compile  accurate  and  complete  models  of  students  within  the  limits  of  their  own 
domain  understanding.  An  example  is  PROUST,  which  is  able  to  infer  possible 
programmer  plans  for  solving  the  single  problem  it  uses  for  all  instructional 
episodes,  computing  average  rainfall  with  a  PASCAL  program.  For  USP-CRmc  to 
achieve  a  similar  capability  would  require  solving  the  plan  recognition  problem,  a 
theme  of  significant  research  interest  in  its  own  right  [Schank,  Abelson  77; 
Schmidt,  Sridharan,  Goodson  78;  London,  Clancey  82;  Carver,  Lesser,  McCue 
84].  Related  efforts  in  goal  infeiencing  is  important  to  dialog  advisoiy  systems; 
the  other  area  where  important  results  in  user  modelling  have  been  achieved. 

3.1.2.  User  Modelling  in  Computer  Advisory  Systems 

User  models  for  advice  giving  systems  based  on  natural  language  dialog 
have  approached  the  user  modelling  problem  from  a  perspective  of  applying  artifi¬ 
cial  intelligence  and  using  linguistics  theory.  A  popular  approach  is  stereotyping; 
it  was  first  proposed  by  Rich  in  the  GRUNDY  system  [Rich  79].  Systems  that  use 
stereotypes  need  other  acquisition  methods  to  first  provide  some  specific  charac¬ 
teristics  about  a  user.  When  the  system  obtains  sufficient  information  about  users, 
it  categorizes  them  as  fitting  a  prestored  stereotype,  and  the  stereotype  then  in- 
direcdy  provides  additional  possible  characteristics.  One  techniques,  used  in  the 
work  on  GRUNDY,  is  to  explicitly  ask  users  for  some  of  these  characteristics.  A 
user-generated  description  aids  the  system  in  selecting  an  appropriate  stereotype. 

Finin  and  Kass  extended  the  stereotyping  approach  to  provide  implicit 
user  model  acquisition  in  a  user  modelling  shell  based  on  a  hierarchy  of  prestoied 
stereotypes  [Kass,  Finin  88a].  Their  systems  analyzes  natural  language  com¬ 
munication  between  the  user  and  the  system  using  the  implicature  rules  adapted 
fiom  Grice’s  Maxims  for  cooperative  communication  [Elass  87a].  An  example  of 
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such  a  rule  \&  If  a  user  says  P,  the  user  modelling  module  can  assume  the  user 
believes  that  P,  in  its  entirety,  was  used  in  reasoning  about  the  current  goal  or 
goals  of  the  interaction.  These  rules,  in  conjunction  with  the  stereotypes,  infer  a 
model  of  the  user’s  goals  and  beliefs.  Chin’s  work  in  KNOME,  a  user  modelling 
component  for  UNIX  CONSULTANT,  used  a  double  stereotyping  technique,  one  for 
grouping  domain  concepts  and  the  other  for  classifying  a  user’s  expertise.  The 
stereotyping  approach  is  useful  for  one-shot  advisory  type  systems  that  need  some 
quick  approximation  of  the  user  in  order  to  quickly  generate  a  piece  of  advice;  it 
could  be  used  as  a  way  to  initializing  user  models  for  critiquing  systems,  if  a  valid 
set  of  stereotypes  is  available. 

Wahlster  and  Kobsa  also  use  the  content  of  a  dialog  to  acquire  a  model 
of  die  user’s  beliefs,  plans,  and  goals  [Wahlster,  Kobsa  88].  Their  work  attempts 
to  emulate  in  a  computer  the  mental  modelling  that  occurs  during  human-to- 
human  communication.  Its  focus  is  insuring  the  system  serves  the  user  in  a 
cooperative  marmer,  as  opposed  to  system  that  might  be  considered  adversarial 
(e.g.,  computer  game-playing  programs,)  or  that  are  at  best  ambivalent  to  the  user 
(e.g.,  express-teller  machines.)  The  user  models  in  this  research  predict  how  a 
user  will  iaierpret  an  utterance  the  system  is  constructing  for  presentation.  In  this 
regard,  the  purpose  of  their  models  are  related  to  Qancey’s  qualitative  model 
framework  for  ICAI  student  models. 

Some  general  characteristics  of  this  class  of  systems  are  quite  different 
than  those  of  cooperative  problem  solving  systems: 

•  The  advice  is  given  in  a  single  episode  and  there  is  no  notion  of  con¬ 
tinuing  dialogs  over  multiple  problems  and  situations.  The  underlying 
asstunption  is  that  the  system  will  never  see  a  user  again  and  if  it  does 
it  will  not  attempt  to  recognize  that  fact  or  use  previous  information 
about  them. 
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•  The  advice  is  generally  atomic;  it  solves  a  given  problem  (e.g.,  invest¬ 
ing  some  money,  locating  an  s^artment,  finding  the  correct  train  to 
reach  a  destination,  etc)  with  a  single  optimal  recommendation. 

•  The  system  is  an  expert.  It  generally  knows  more  about  the  advisory 
domain  than  the  user;  and  an  implicit  assumption  is  that  what  the 
computer  advisor  recommends  is  accepted  without  question  as  being 
appropriate  and  optimal. 

•  The  system  is  not  concerned  with  supporting  users’  learning  in  the  ap¬ 
plication  domain.  Its  goal  is  to  insure  the  advice  is  understood  with  an 
assumption  that  once  users  understand  what  is  being  suggested  they 
willingly  accept  recommendation. 

Cooperative  problem  solving  requires  that  systems  be  prepared  to  deal  with  the 
same  user  repeatedly,  and' do  so  in  domains  where  a  conq>lex  product  is  being 
produced.  Furthermore,  it  will  be  the  case  that  both  parties  share  responsibility  for 
the  result  and  each  have  some  knowledge  to  contribute  to  the  solution  process. 
Cooperative  problem  solving  systems  will  therefore  need  models  that  are  dynamic, 
persistent,  and  idiosyncratic. 

An  important  distinction  in  purposes  for  user  models  is  important.  User 
models  can  help  a  system  to  generate  the  q:)propriate  suggestions  (for  our  systems 
these  are  in  the  form  of  the  critique,  for  advisory  systems  a  recommended  course 
of  action),  or  in  a  general  sense  help  explain  some  facet  of  the  domain.  Specifi¬ 
cally,  for  critics  and  advisory  systems,  that  is  an  explanation  of  the  rationale  be¬ 
hind  the  suggestion  in  terms  of  domain  concepts.  Ideally  the  same  user  model  will 
serve  both  purposes.  Research  in  advisory  system  has  focused  on  the  first  situa¬ 
tion  —  insuring  the  advice  is  appropriate  to  user  goals  and  plans,  while  the 
research  here  focuses  on  the  second  purpose  —  explaining  to  the  user  the  domain 
knowledge  underlying  a  given  critique. 
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To  summarize,  there  are  several  significant  differences  between  user 
modelling  for  critiquing  and  those  that  support  advisory  dialog  systems.  The  no¬ 
tion  of  a  product  constructed  through  a  collaborative  effort  between  the  system  and 
the  user  is  central  to  most  critics.  Advisory  systems  are  designed  as  all-knowing 
experts  which,  once  they  infer  sufficient  information  about  the  user,  will  select  or 
generate  proper  advice.  The  user’s  role  is  passive  while  in  critics  both  the  system 
and  the  user  are  active  in  solving  the  problem  at  hand.  Advisory  systems 
predominantly  exercise  control  of  the  human-computer  interaction.  They  are  less 
"system  controlled"  than  intelligent  tutoring  systems,  but  overall  responsibility  for 
the  interaction  resides  in  the  system.  In  critics,  the  system  and  the  user  share 
responsibility  for  solving  the  problem  at  hand  and  for  guiding  the  interaction.  Like 
in  the  student  modelling  work  there  are  several  techniques  developed  by  research 
in  this  area  that  can  potentially  be  integrated  into  a  framework  for  models  that 
support  cooperative  problem  solving;  they  include:  stereotyping  approaches,  the 
distinction  between  explicit  and  implicit  acquisition  techniques,  and  inference 
rules  that  use  the  content  of  the  human  computer  dialog  to  enridi  the  user  model 
contents.  These  together  with  key  ideas  from  the  student  modelling  work  guided 
the  articulation  of  some  foundations  for  the  approach  followed  in  this  dissertation 
work. 

3.2.  Foundations  for  User  Models  to  Support  Cooperative  Problem  Solving 

There  are  three  issues  that  need  to  be  addressed  for  user  modelling  in 
cooperative  problem  solving  system:  how  to  represent  the  user  model,  how  to  ac¬ 
quire  it,  and  how  to  access  it.  The  first  two  areas  proved  to  be  the  most  difficult; 
access  of  the  models  is  primarily  determined  by  decision  about  the  representation. 
The  acquisition  problem,  viewed  in  the  ITS  literature  as  a  problem  of  diagnosis,  is 
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the  most  challenging.  To  synthesize  my  review  of  the  research  literature  con¬ 
cerned  with  user  modelling,  a  topology  was  used  to  summarize  the  work.  It 
categorizes  specific  ideas  and  projects  into  the  areas  of;  the  knowledge  the  user 
model  represents,  how  it  is  acquired,  and  its  primary  purpose.  Appendix  A  con¬ 
tains  a  table  showing  the  systems  discussed  throughout  this  dissertation.  Their 
characteristics  in  each  category  together  with  their  qiplication  domains  and  the 
purpose  of  the  systems  themselves  are  listed. 

In  the  area  of  acquisition  techniques,  I  found  it  useful  to  categorize  them 
based  on  the  directedness  of  the  inferencing  method. 

1.  Direct  acquisition  techniques  are  those  where  a  specific  piece  of 
information  is  obtained  by  explicitly  questioning  users  or  from 
implicit  observations  of  them.  Usually  a  single  characteristic  about  a 
user  is  inferred. 

2.  Indirect  acquisition  techniques  are  shortcuts,  such  as  stereotypes  or 
classification  schemes;  they  are  always  implicit. 

In  the  literature,  the  more  commonly  used  distinction  for  acquisition  techniques, 
(described  best  in  [Kass,  Finin  87a]),  is  implicit  versus  explicit  acquisition  ap¬ 
proaches,  the  orthogonality  of  these  two  categorizations  is  shown  in  Table  3-1. 

Table  3-1:  Two  Orthogonal  Qassifications  of  Acquisition  Techniques 


Categorizing  User  Model  Acquisition  Techniques 

Direct  Techniques 

Indirect  Techniques 

Implicit  Acquisition 

X 

X 

Explicit  Acquisition 

X 

The  user  characteristics  represented  in  the  model  make  a  claim  about 
what  users  can  do;  what  they  know;  and  their  goals,  plans,  prejudices  or 
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preferences.  To  support  the  first  two  types  of  information,  the  representation  must 
be  in  terms  of  domain  expertise.  Users  do  not  simply  know  or  not  know  a  skill  or 
domain  entity,  so  representing  their  knowledge  using  a  binary  value  is  inadequate. 
Research  in  some  student  models  tackle  this  problem  by  attempting  to  rate  the 
knowledge  of  each  domain  entity  in  the  user  model  with  a  linear  value.  A  linear 
coefficient  used  to  represent  the  degree  of  proficiency  would  be  ideal,  but  the  dif¬ 
ficulty  is  that  to  establish  the  validity  of  such  coefficients  requires  extensive  statis¬ 
tical  analysis  of  the  population  of  users.  Prevailing  approaches  have  used  ad  hoc 
methods  for  setting  these  values.  Th^  research  usually  is  oriented  on  demonstrat¬ 
ing  how  the  acquisition  process  works  rather  than  evaluating  the  validity  of  the 
models  themselves.  A  simple  approach  is  to  represent  each  user  according  to  a 
classification  of  domain  expertise  (e.g,  expert,  novice,  beginner).  This  is  what  I 
call  a  “classification  method”;  it  is  an  approach  which  can  be  viewed  as 
analogous,  or  even  derived  from,  the  stereotyping  methodology.  In  this  project,  an 
alternative  method  for  the  system  to  categorize  how  well  a  user  knows  some  piece 
of  domain  knowledge  was  needed. 

3.2.1.  Classifying  the  Users’  Domain  Knowledge 

In  [Fischer  88a]  such  a  schema  for  classifying  users’  knowledge  was 
presented,  it  is  shown  graphically  in  Figure  3-1.  This  schema  provides  a  basis  for 
the  user  modelling  component  developed  in  this  research.  It  provides  a  conceptual 
model  for  the  space  of  user  knowledge  in  the  application  domain.  In  general,  the 

domains  in  the  figure  represent  the  following: 

D|:  The  subset  of  concepts  (and  their  associated  commands)  that  users  know  and  use 
without  any  problems. 

D2:  The  subset  of  concepts  which  they  use  only  occasionally,  users  do  not  know 
d^ails  about  them  and  are,  possibly,  unsure  of  their  effects. 

D3:  The  mental  models  [Norman  82;  Fischer  84]  of  the  users,  i.e.,  the  set  of  concepts 
which  they  think  exist 

D^:  This  region  represents  the  actual  set  of  concepts  in  of  a  domain. 
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A  specific  inteipretation  of  this  model  in  terms  of  the  domain  our  user  model 


serves,  LISP,  will  be  offered  in  Chapter  7. 


Figure  3-1:  Levels  of  System  Usage 


Using  this  schema  as  a  basis  for  the  user  model  representation  means 
that  it  is  necessary  to  capture  how  well  a  user  understands  domain  entities  in  ac¬ 
cordance  with  these  levels.  The  level  at  which  users  know  a  domain  entity  can,  in 
turn,  guide  explanation  giving;  this  will  be  shown  in  Cluster  6. 

3.2.2.  General  Approaches  to  User  Modelling 

Human-computer  interaction  includes  many  different  types  of  systems 
and  interaction  t^roaches.  A  common  theory  for  how  to  design  and  ^>ply 
idiosyncratic  user  models  across  different  areas  is  desirable  because  it  will  allow 
sharing  of  research  results  and  identification  commonly  usable  features  in  in¬ 
tegrated  systems  —  system  that  use  more  than  one  approach  to  interaction. 

Establishing  the  requirements  for  general  user  modelling  can  be  pursued 
in  two  different  ways.  One  strategy  is  framework-driven:  it  defines  a  common 
architecture  that  can  be  used  by  any  system.  The  General  User  Modelling  Facility 
(GUMS)  [Kass,  Finin  88a]  provides  such  a  framework.  It  is  a  top-down  approach 
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because  the  framework  is  conceptually  predefined  and  can  guide  research  as  well 
as  development  efforts  for  specific  systems,  domains,  or  paradigms.  Another  at¬ 
tempt  at  developing  a  domain-independent  modelling  subsystem  is  the  “User 
Modeling  Front  End”  (UMFE)  [Sleeman  84].  A  conunon  idea  with  this  work  is 
the  specification  of  sets  of  inference  niles  based  on  diagnostic  information  about 
how  user’s  knowledge  propagates  through  a  set  of  concepts.  These  generalized 
modelling  approaches  attempt  to  encapsulate  a  complete  theory  of  user  modelling 
that  could  be  qTplied  to  any  system. 

A  more  system-driven  approach  is  a  bottom-up  strategy,  studying  “suc¬ 
cessful”  user  modelling  systems  in  different  domains  and  paradigms,  then  reusing 
appropriate  techniques  and  ideas  in  the  user  modelling  component  of  a  specific 
system.  One  example  of  this  is  the  overlay  modelling  technique  first  proposed  in 
WUSOR-n,  it  has  become  a  standard  ITS  paradigm  [Wenger  87].  Another  is  the  use 
of  bugs  to  perform  student  diagnosis  and  repair  in  systems  such  as  BUGGY  and 
DEBUGGY  [Brown,  VanLehn  80],  and  the  Leeds  Modelling  System  (LMS) 
[Sleeman  83];  then  later  applied  to  other  domains  such  as  programming  [Gray, 
Corbet,  VanLehn  88]. 

Common  features  of  successful  models  can  be  used  to  drive  tneoretical 
developments  in  the  field;  it  is  a  case  of  the  system  implementation  and  testing 
work  driving  the  development  of  a  general  approach  or  theory.  This  dissertation 
has  primarily  embraced  this  approach.  A  methodological  first  step  toward 
developing  a  user  modelling  approach  for  critiquing  systems  is  to  build  a  system 
based  on  both  what  we  understand  to  be  the  system’s  needs  and  integrating  good 
ideas  from  other  research.  Theories  need  to  be  tested  by  developing  systems,  and 
system  implementations  need  to  be  studied  to  refine  the  theory.  The  work  here  has 
concentrated  first  on  selecting  worthy  techniques  from  other  user  modelling  areas 
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and  integrating  them  into  a  proposed  theoretical  framework.  That  framework  was 
enhanced  during  implementation  of  a  user  modelling  component  for  a  computer- 
based  critic.  Over  the  long  term,  that  implementation  should  drive  additional 
theoretical  research  to  provide  a  theory  of  user  modelling  to  support  not  just 
cooperative  problem  solving  but  a  larger  class  of  interactive  systems  that  includes 
tutoring  and  advising,  amonr  others.  The  requirements  placed  upon  a  user  model 
for  systems  that  support  of  cooperative  human-computer  effort  are  discussed  next. 

3.2.3.  Requirements  for  User  Models  in  Cooperative  Problem  Solving  Systems 

Communication  is  at  the  heart  of  any  cooperative  effort.  In  order  for  a 
human  and  computer  to  collaborate  effectively  they  must  communicate  about  the 
product,  perhi^s  the  goal,  and  general  information  about  the  domain  in  the  form  of 
computer-produced  explanations.  Explanations  in  the  systems  we  are  investigat¬ 
ing  are  currently  uni-direaional,  from  the  computer  to  the  user.  In  the  future  an 
plication  of  machine  learning  research  might  be  for  users,  ta  also  explain  their 
knowledge  to  the  computer  as  a  way  for  the  system  to  leani  more  about  the 
domain.  In  either  situation,  dialogs  between  the  system  and  user  need  to  operate  at 
a  level  commonly  understood  by  both  agents. 

The  user  model  needs  to  be  accessible  to  other  system  components.  Its 
contents  will  be  used  when  presenting  explanations,  selecting  items  to  analyze, 
and  perhaps  as  a  record  of  user  preferences  used  to  tailor  the  system.  The  ultimate 
objective  is  an  integrated  system  which  adapts  to  users,  allows  them  to  specify 
preferences,  and  is  still  somewhat  consistent  in  the  way  it  treats  them. 

The  user  models  in  cooperative  problem  solving  systems  will  have  to  be 
more  individualized  than  those  provided  in  classification  schemes  or  stereotyping 
^jproaches.  Users  of  a  complex  system  are  not  homogeneous  and  the  system 
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needs  to  treat  each  one  differently.  Having  individual  models  alone  is  inadequate, 
their  contents  have  to  change  as  the  individuals  knowledge  in^roves  —  users 
usually  become  more  knowledgeable  or  proficient  over  time.  A  precept  of 
cooperative  problem  solving  is  to  provide  an  environment  that  serves  users  not 
once  but  on  a  recurring  basis;  this  means  the  system  adapts  and  changes  as  users 
change.  Achieving  system  adaptivity  requires  a  representation  of  each  user  that  is: 

•  dynamic  —  it  changes  over  time, 

•  persistent  —  it  is  retained  between  problem  solving  episodes  and 
reused  by  the  system,  and 

•  idiosyncratic  —  it  is  unique  for  each  individual. 

Based  on  these  requirements,  several  ideas  from  the  analysis  of  related 
research  on  user  and  student  modelling  were  identified  for  incorporation  into  a 
user  modelling  framework  for  cooperative  problem  solving  systems: 

1.  Stereotyping  (GRUNDY) 

2.  Explicit  and  implicit  acquisition  methods  (GUMS) 

3.  Representing  user  knowledge  as  a  perturbation  of  the  domain 
(Genetic  Graphs) 

4.  Using  the  dialog  content  as  a  basis  for  acquisition  inferencing 
(GUMAC) 

5.  Acquisition  methods  based  on  the  relationship  between  knowledge, 
the  structure  of  the  domain  model  (UMFE) 

The  architecture  for  that  user  modelling  component  will  be  covered  next. 

3  J.  A  User  Model  Architecture 

A  conceptual  architecture  for  the  user  modelling  component  that  is 
derived  from  the  previously-discussed  requirements  was  developed.  That  architec- 
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tuie  is  designed  to  serve  the  needs  of  the  specific  system  and  critiquing  paradigm 
but  with  an  eye  toward  retaining  suMcient  generality  that  it  might  serve  as  the  user 
modelling  component  for  any  cooperative  problem  solving  system.  There  are 
three  major  subcomponents  of  the  architecture,  the  representation  scheme,  acquisi¬ 
tion  techniques,  and  access  methods;  they  are  shown  in  Figure  3-2, 


Figure  3-2:  General  Architecture  for  A  User  Modelling  Component  for  CPSS 


3  J.l.  Representation 

The  representation  scheme  is  central  because  it  must  support  acquisition 
and  access.  It  must  also  be  general,  efficient,  and  easy  to  expand  or  modify.  An 
additional  consideration  is  to  make  the  schema  understandable  to  a  human  or,  at 
least,  able  to  be  presented  by  the  system  in  a  forni  that  humans  can  read  and 
modify.  We  woiild  like  for  the  modelling  component  to  support  either  users  them¬ 
selves  or  a  teacher  in  interpreting  and  editing  individual  models.  User  models  are 
at  best  approximate  representations  of  some  cognitive  aspects  of  an  individual  and 
we  should  allow  for  those  situations  where  that  individual  or  another  human  can 
improve  on  that  approximation. 
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The  two  ideas  from  other  research  in  user  modelling  that  contribute  to 
our  representation  scheme  are:  the  use  of  a  grr^h  model  for  the  domain,  such  as 
the  genetic  graph,  and  representing  the  user  as  an  overlay  of  the  domain  model. 
The  implementation  we  developed  uses  a  more  general  approach  than  genetic 
graphs  to  represent  the  domain  and  a  coloring  of  those  graphs  that  is  based  on  the 
schema  for  representing  user  knowledge. 

3  J.2.  Acquisition 

Acquiring  the  user  model  is  the  most  complex  function  in  this 
architecture.  It  requires  knowledge  on  the  part  of  the  system,  knowledge  about 
ways  to  infer  the  state  of  the  user.  Representing  and  accessing  a  user  model  could 
be  achieved  using  common  database  techniques  if  acquisition  was  not  such  a  corn* 
plex  problem. 

The  acquisition  methodology  will  need  to  support  various  approaches 
for  acquiring  information  about  a  user.  The  collection  of  acquisition  techniques  in 
the  system  can  be  conceptually  viewed  as  a  knowledge-based  agent;  an  agent  that 
is  able  to  infer  what  a  user  knows  from  information  provided  by  other  system  com¬ 
ponents  which  track  the  human-conq^uter  dialog;  the  agent  also  knows  explicit 
questions  to  ask  that  help  infer  the  model  contents,  or  certain  stereotypes,  etc. 
Four  categories  of  possible  acquisition  ^>proaches  were  identified: 

•  Explicit  techniques  directly  question  the  user  for  information  that  is 
entered  into  the  user  model.  It  is  a  suitable  approach  for  obtaining  an 
initial  user  model  as  it  can  be  implemented  as  a  simple  up-front  ques¬ 
tionnaire  or  testing  session  when  a  user  accesses  a  system  for  the  first 
time.  It  is  not  as  suitable  during  subsequent  human-system  interaction 
episodes  because  users  are  not  willing  to  put  up  with  such  administra- 
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tive  requirements  more  thtn  once.  If  a  model  so  acquired  is  not 
changed  to  reflect  changes  to  users’  knowledge  it  wiU  become  con¬ 
tinuously  less  valid  and  useful  —  the  approximation  of  the  user’s 
knowledge  state  becomes  progressively  less  approximate. 

•  Implicit  techniques  enrich  the  user  model  without  interrupting  the 
user.  Two  implicit  techniques  are  of  interest:  stereotyping  and  the 
implicit  implicature  rules  that  operate  on  the  human-computer  dialog. 
Stereotypes  are  difficult  to  apply  in  many  situations  mainly  because, 
as  discussed  earlier,  it  is  hard  to  determine  what  stereotypes  to  use. 
Organizing  those  stereotypes  into  a  hierarchy  presents  its  own 
problems  [Kass,  Finin  87b].  Some  implicature  rules,  as  will  be 
described  in  Chapter  7,  can  be  modified  so  that  they  apply  to  the 
human-computer  dialog  present  in  most  con:q>uter  working  environ¬ 
ments  rather  than  natural  language  situations  alone.  There  are  also 
implicit  techniques  that  are  indirect;  they  use  the  domain  model  struc¬ 
ture  to  leverage  the  infonnation  provided  by  inq}licature  rules. 

•  Tutoring  methods  acquire  information  from  instructional  episodes  that 
can  be  added  to  or  used  to  modify  user  models.  These  are  episodes 
initiated  either  by  user  request  or  by  the  system  for  the  express  pur¬ 
pose  of  evaluating  a  user’s  knowledge.  There  are  not  any  systems  that 
attempt  to  do  the  latter  but  this  appears  to  be  a  natural  combination  of 
ideas  in  tutoring  and  user  model  acquisition  worthy  of  investigation. 
Information  in  the  model  that  appears  to  be  missing  or  in  conflict  trig¬ 
gers  a  tutoring  episode  in  which  the  system  poses  a  problem  to  the 
user,  one  designed  or  selected  to  evaluate  user  understanding  of  the 
knowledge  in  question.  Given  a  comprehensive  system,  such  as 
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GRACE  [Dews  89;  Atwood  et  al.  90]  that  combines  both  a  critic  and  a 
tutor,  if  a  user  voluntarily  requests  some  tutoring,  whatever  subjects 
are  addressed  during  that  tutoring  can  be  used  by  the  system  in  a 
similar  fashion. 

•  Statistical  user  model  acquisition  methods  could  be  included  in  the 
implicit  category  but  they  ate  of  stifficient  interest  to  warrant  their 
own  separate  category.  The  acquisition  technique  is  one  of  observing 
user  actions,  accumulating  a  history  of  those  actions  (usually  in  the 
form  of  a  count),  and  triggering  inference  methods  upon  reaching 
predefined  threshold  levels  in  that  stadsticaL  history.  The  thresholds 
trigger  an  inference  about  the  user  and  precipitate  an  offer  of  critiqu¬ 
ing  type  advice  to  the  user.  In  the  ACTIVIST  system  models  based  on 
statistical  methods  proved  to  be  effective  [Fischer,  Lemke,  Schwab 
84].  Unfortunately,  this  ^roach  has  not  been  explored  except  in  that 
research,  and  only  conceptually  in  LlSP-CRmc. 

33.3.  Access 

The  access  methods  are  the  third  part  of  the  architecture.  The  model 
contents  must  be  accessible  and  usable  by  other  components,  or  perhaps  human 
agents.  Access  methods  provide  information  to  other  system  components  about 
what  the  user  does  or  does  not  know  about  the  domain.  In  the  model  developed  in 
this  research  that  access  provides  information  to  guide  explanations,  but  the 
methods  are  generic  in  nature  so  that  they  could  support  the  needs  of  tutoring, 
advisors  and  so  forth.  Access  functions  need  to  be  general  enough  to  support 
known  requirements,  and  flexible  to  accommodate  extensions  to  the  system.  Ac¬ 
cess  methods  are  not  conceptually  or  theoretically  difficult  but  are  most  often 
determined  by  the  language  or  methodology  used  to  implement  the  representation. 
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3.4.  Summary 

A  general  firamewoik  for  a  user  modelling  component  enable  of  sup¬ 
porting  cooperative  problem  solving  was  developed  in  this  chq)ter,  it  incoiporates 
techniques  from  research  on  student  modelling  in  intelligent  computer-aided  in¬ 
struction  and  user  modelling  in  advisory  dialog  systems.  The  initial  strategy  was 
to  select  a  technique  from  one  of  these  areas  that  could  be  modified  to  meet  the 
needs  of  cooperative  knowledge-based  systems.  However,  such  a  direct  plica¬ 
tion  was  not  feasible  because  the  theoretical  analyses  showed  that  there  are  suf¬ 
ficient  differences  in  the  needs  of  the  different  types  of  systems  in  terms  of  what 
they  need  to  know  about  their  users,  and  in  the  control  of  the  interaction.  Alter¬ 
natively,  an  architectural  framework  for  a  user  model  was  specified;  one  that  is 
able  to  support  cooperative  human-computer  effort,  is  based  on  a  categorization  of 
users*  expertise,  and  is  general  in  nature.  That  conceptual  framework  has  been 
instantiated,  in  part,  in  a  user  modelling  component  that  will  be  described  in  Chap¬ 
ter  7.  The  implementation  context  is  LiSP-CRinC;  die  next  Clpter  provides  an 
overview  of  how  LISP-Crttic  has  evolved  over  time,  how  it  is  currently  con¬ 
figured,  and  how  it  presendy  operates. 


CHAPTER  IV 


LiSP-CRmc 

LiSP-CRTnc  was  used  as  the  development  environment  in  which  the  user 
modelling  framewoik  was  implemented.  It  is  a  knowledge-based  system  that  is 
designed  to  support  programmers  in  the  context  of  their  work.  It  does  not  have 
‘‘automatic  programming”  capabilities  but  operates  according  to  same  principle 
of  ‘‘intelligent  assistance”  that  is  fundamental  in  the  PROGRAMMER’S 
APPRENTICE  work  (Rich,  Waters  90].^  In  the  terms  of  that  research  LiSP-CRmc 
belongs  to  the  class  of  what  are  called  ‘‘transformation  system”  [Rich,  Waters 
88]. 

Comparisons  between  LISP-CritiC  and  the  work  on  LISP  Tutor  are  in¬ 
evitable.  As  discussed  in  the  context  of  learning  enviroiunents  in  Ch^ter  1,  the 
purposes  for  the  two  systems  ate  actually  quite  diverse.  LISP  Tutor  proposes  to 
teach  the  LISP  programming  by  leading  students  through  a  series  of  predetermined 
programming  exercises  known  to  the  system  in  detail.  USP-CRTIIC  is  oriented 
toward  aiding  programmers  involved  in  real  work  by  suggesting  to  them  better 
ways  to  implement  a  specific  piece  of  code  they  have  written.  In  LiSP-CRmC,  like 
in  any  critiquing  system,  learning  will  inevitably  occur,  but  it  would  be  best  to 
incorporate  capabilities  into  the  system  to  make  that  learning  as  effective  as  pos¬ 
sible. 

^The  long  term  vision  for  the  PROGRAMMER’S  APPRENTICE  is  that  it  ‘‘aa  as  a 
software  engineer’s  junior  partner  and  critic  (emphasis  added)”  [Rich,  Waters  90, 
p.  1].  In  our  view,  envelopment  of  LISP-Crttic  provides  significant  understanding 
of  what  is  involved  in  the  critic  portion  of  such  a  system. 
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LlSP-CRmc  provides  a  suitable  context  for  investigating  both  user 
modelling  and  the  cooperative  problem  solving  paradigm  for  several  reasons: 

•  The  rule  knowledge  base  in  LISP-Crttic  was  previously  developed 
and  has  been  refined  through  several  versions  of  the  system,  therefore 
this  research  did  not  have  to  contend  with  acquiring  and  testing  the 
executable  knowledge  in  the  system. 

•  Critiquing  is  a  paradigm  that  has  been  studied  and  is  well  understood, 
as  discussed  in  Chapter  2;  therefore,  we  could  consider  extending  it, 
in  the  context  of  LlSP-CRmc,  it  to  integrate  user  models  and  support 
cooperative  human-computer  work. 

•  The  part  of  the  process  involved  in  giving  a  programmer  advice  (the 
initial  critique  or  suggestion  that  we  will  see  in  the  scenario)  is  stable 
and  usable.  This  is  due,  in  part,  to  the  maturity  of  the  rule-based 
knowledge. 

The  system  was  not  built  from  scratch  for  this  project;  it  has  been  the 
focus  of  iterative  development  over  several  years.  This  chapter  reviews  the  dif¬ 
ferent  versions  of  LlSP-CRmC  and  some  specific  research  projects  to  enhance  the 
system  that  in  part  motivated  this  work.  It  will  then  describe  the  current  version  in 
terms  of  its  architecture  and  will  use  a  scenario  of  a  user  interacting  with  the  sys¬ 
tem  to  demonstrate  specific  points.  Those  portions  of  the  current  system  central  to 
this  project,  the  domain  model,  explanation  giving,  and  the  user  modelling  com¬ 
ponent  are  described  in  more  detail  in  Chapters  5, 6  and  7,  respectively. 

4.1.  Lineage  of  Lisp-Critic  Versions  and  Research  Issues  Addressed 

LiSP-CRmc  has  evolved  from  a  knowledge-based  code-enhancement 
tool  to  a  programming-design  environment.  In  that  process,  it  has  benefited  from 
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the  integration  of  interactive  capabilities,  contextual  critiquing  and  explanation 
capabilities;  all  help  to  evolve  the  system  toward  one  which  meets  the  theoretical 
notions  of  being  a  cooperative  problem  solving  system.  The  system  has  existed  as 
four  distinct  versions  (see  Figure  4-1,)  each  one  using  ideas  and  parts  from  the 
previous  version,  but  improving  on  them,  and  integrating  new  ideas.  Its  system 
development  history  is  similar  to  the  series  of  mutation  and  selection  steps  found 
in  genetic  evolution.  It  exemplifies  the  Simon  view  of  evolutionary  software 
development  [Simon  81]. 

Modifications  made  in  producing  each  new  version  addressed  new 
research  issues;  these  are  indicated  with  ovals  in  Figure  4-1.  Each  version 
generated  an  enhanced  conceptual  model  of  the  system,  and  contained  new  or  im¬ 
proved  parts  based  on  what  was  learned  in  developing,  using,  and  evaluating  the 
previous  versions.  The  first  three  systems  will  be  discussed  briefly,  followed  by  a 
description  of  the  current  version.  None  of  the  versions  discussed  here  was  in¬ 
tended  to  be  a  commercial  product  or  even  a  full  prototype  for  general  use,  rather 
they  are  more  in  the  spirit  of  what  Rich  &  Waters  call  “demonstration  systems”. 
They  were  developed  as  a  context  in  which  to  investigate  theoretical  issues, 
hypothesize  solutions,  and  implement  the  solutions  to  demonstrate  how  they  work 
and  gain  additional  insight  that  was  used  to  refine  the  concepts. 

CODElMPROYER.  The  precursor  to  LISP-CRITIC  was  the 
CODEiMPROVER  system  [Boecker  84].  CODEiMPROVER  is  a  knowledge-based 
program  transformation  system.  Once  invoked  it  operates  independently,  not  al¬ 
lowing  further  user  interaction.  Input  to  CODE-IMPROVER  is  an  executable 
FRANZUSP  program  and  the  output  is  a  version  of  that  program  that  either  better 
supports  human  understanding,  one  that  is  more  cognitively  efficient,  or  a  version 
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Vsrsim^  of  LlSP-Crifte 


The  versions  of  LlSP-CRlTIC  are  shown  in  the  center  of  the  above  figure. 
Each  version  addressed  new  theoretical  issues  shown  in  the  ovals. 

Figure  4-1;  Theoretical  Issues  Addressed  in  Versions  of  LlSP-CRmc 


that  makes  better  use  of  computing  resources,  one  that  is  more  machine  efficient. 
The  transformations  used  by  the  system  are  captured  in  a  rule  base  that  was 
developed  using  traditional  knowledge  acquisition  approaches;  these  rules  were 
elicited  from  expert  programmers  through  interviews.  An  example  of  the  sort  of 
rules  contained  in  that  knowledge-base  is  shown  in  Figure  4-2.  CODE  IMPROVER 
operates  in  a  batch  mode  in  the  UNIX  operating-system  environment,  reading 
programs  and  then  writing  an  improved  version  of  them  into  user  files. 
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Replace  a  Copying  Function  with  a  Destructive  Function 


(rule  append/ . 1-new. cons . cella-to-nconc/ . 1 . . . 
( ?  f oo : { append  appendl ) 

(restrict  ?expr 

(cons-cell-generating-eacpr  expr) ) 


?b) 

( (con^ute-it ; 

(cdr  (assq  (get-binding  foo) 

' ( (append  .  nconc) 

(appendl  .  nconcl) ) ) ) ) 

?expr  ?b) 
safe  (machine) ) 

(append  (explode  word)  chars) 

(nconc  (explode  word)  chars) 


;;;  the  name  of  the  rule 
;;;  the  original  code 
;;;  condition 
;;;  (only  apply  rule 
if  ?expr'' generates 
;;;  cons  cells) 


;;;  the  replacement 


rule  category 


The  rale  ** appendl. fnew. cons. cell-to-nconc”  replaces  the  function 
APPEND,  which  generates  a  copy  of  its  argument  data  structure  in 
memory,  with  the  function  NCONC  which  instead  modifies  the  internal 
representation.  The  latter  is  preferred  when  users  want  to  minimize 
memory  use  and  the  new  data  structure  is  not  needed  elsewhere  in  the 
program. 

Figure  4-2:  Example  of  a  Rule  in  LiSP-CRmc 


WLiSP  Version  The  first  version  actually  called  LlSP-CRmc  [Fischer 
87b]  was  designed  to  nm  in  the  WLiSP  windowing  envirorunent  [Fabian,  Lemke 
85]  on  BITGRAPH  terminals.  It  provides  some  rudimentary  explanation  capability 
of  the  critic’s  suggestions  by  showing  what  rules  were  fired.  Users  can  choose  the 
land  of  suggestions  in  which  they  are  interested.  This  version  was  designed  to 
take  advantage  of  advances  in  human-computer  interaction  techniques  (such  as 
windowing  environments,  menus,  and  the  mouse)  and  to  enhance  learning. 
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Lisp  Machine  Version,  order  to  bring  LiSP-CRmc  closer  to  support¬ 
ing  Lisp  programmers  in  their  current  working  situation,  it  was  integrated  into  a 
USP  Machine  environment,  the  Symbolics  3600  Workstation.  This  version  was  a 
direct  precursor  to  the  work  reported  here.  Integrating  LiSP-CRmc  with  the  other 
functionalities  of  the  Symbolics  Genera  environment  provided  a  better  understand¬ 
ing  of  the  capabilities  and  limitations  of  critiquing.  When  the  system  was  able  to 
make  use  of  an  environment  that  provides  powerful  interface  capabilities,  like 
those  available  on  the  Symbolics,  this  changed  our  view  of  what  to  expect  from 
the  system,  and  how  to  configure  its  architecture.  Figure  4-3  shows  that  second 
version  of  LlSP-CRmc  running  as  an  activity  in  the  Genera  Environment.  The 
knowledge  base  of  USP-CRmc.  was  updated  to  process  COMMON  LISP  but  the 
form  of  knowledge  it  contains  and  the  way  it  applies  that  knowledge  did  not 
change  from  previous  versions. 

Several  ideas  were  tested  in  this  version  that  were  designed  to  make  the 
environment  more  interactive.  Some  of  c^abilities  that  were  provided  to  users 
were: 

•  to  view  and  compare  the  two  versions  of  the  program  —  their  original 
code  and  the  one  generated  by  USP-CRTTIC  (shown  in  code  pane  1 
and  code  pane  2,  respectively), 

•  to  request  explanation  of  the  differences  between  the  two  versions, 

•  to  invoke  LISP-CRITIC  on  source  code  files  in  any  local  or  remote 
directory,  and 

•  to  have  use  of  the  inteireferential  input/output  features  in  the  Genera 
environment. 

Explanations  were  provided  in  the  form  of  rule-traces,  like  in  the  MYCIN 
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LISP-Cricic  [version  1. 3] 


LISf-CrItU  rwlcs  Hlileli  find 
•hOMn  in  Uie  foUOMing  fornati 


Mtt  »“»M»r9»»i0n  frm  your  cotfo 


n«A«  of  LlS^-Critic  rulo  vhteft  find 


tho  trontfornod  a~oMproooion 


To  M*  on  •Molonotien  for  ony  of  th* 
rules,  ut«  nsnu  eetlon  oMptoin  nlo 


(a  «>  jtmtn  nt*c  <1i*t  ■»  i.vt  <«M  n  y)>  w)  v)) 

<ear  i)  tpmor  (ear  <)» 

C1*t  tC«  Cw  A1 

<w  IpoMOT  («Sr  •)»> 

<li«l  (MM  a))  •'((«*«•  (y>  (Mna  a  y»  v»  w)> 


C(<  »)  r)  •Ml) 

Ct  (MOcar  1‘OMaB  (v>  («M  (car  u)  v)>  (cap*  (asr  u)  (t-*  r))») 

Rwici  f  rt  to'at^S  •••%► 

(t«  (fwii  (<  it)  p»  (i^car  t'dwM*  (y>  (esM  (car  u)  y»  (asi*  (c«r  w) 

tl-  r)))> 


<(•  p  S)  Ml)  (t  (CM  (car  0  (BCO  (ear  t>  Cl*  p)»>) 

Ici  oanaHa^v^S 

(tf  («apil  (•  r  e»  (capw  (aar  a)  (pc«  taar  a)  (i-  r))» 


CImt  Oisoliy  Cxolam  Ruts  Optimizs  Shoi#  Ruiss  Fired  Simplify  File 

Wepiey  Oirectery  Help  Redisplay  Code  Stmpfify  expression 


rlttc  oopinendi  SinpUfy  flunCH:  >thonaan>ooMer •  Map 
LlSP^rftic  eewwsnd(  Show  Rules  Firee 
LlSP'>Cr1tie  eonwendi  EMpleln  Rule 
LlS^'Crltic  oonrendt  EKpleln  Rule 
U6P-Cr1tle  eomiend:  Explain  Rule 


This  is  LISP-CritiC’s  screen  on  the  Symbolics  3600.  Users  can  request  a 
critique  of  a  program  code  file  using  the  menu  options  or  can  enter  a  LISP 
expression.  They  receive  suggestions  on  to  how  to  improve  that  code  in 
CODE  PANE  2,  their  own  code  is  show  in  code  pane  1.  Rule  tracing  ex¬ 
planations  of  LiSP-CRmC’S  suggestions  are  available.  In  the  Figure,  the 
user  submitted  a  program  for  critiquing,  has  seen  a  trace  of  the  rules  and  is 
about  to  select  an  explanation  for  one  of  those  rules  from  a  pop-up  menu. 


Figure  4-3:  LiSP-CRmc  Interface  on  the  Symbolics  Computer 


[Buchanan,  Shortliffe  84]  and  subsequent  GUIDON  [Qancey  87]  research.  If  fur¬ 
ther  clarification  is  required,  the  system  presents  a  pre-stored  textual  description  of 
a  rule,  a  description  that  is  general  in  nature,  not  specific  to  the  suggested  transfor¬ 
mation.  This  generic  explanation  approach  was  one  of  the  shortcomings  in  this 


version. 


During  development  of  this  version,  the  system  was  evaluated  by  two 
different  user  groups.  Intermediate  users  want  to  leam  to  produce  better  LISP 
code;  for  supporting  this  purpose,  statistical  data  were  gathered  coneeniing  the  fre¬ 
quency  of  rules  that  fired  in  student  programs.  Another  group  of  experienced 
users  want  to  “straighten  out”  their  code.  Instead  of  refining  a  program  by  hand 
(which  in  principle  they  are  capable  of  doing),  they  use  LiSP-CRmc  to  cause  them 
to  reflect  on  the  design  decisions  they  made  and  the  code  produced  to  inclement 
them.  The  critiquing  rqiproach  is  especially  useful  for  improving  code  that  is  ei¬ 
ther  under  development  or  frequently  modified.  In  the  context  of  these  develop¬ 
ment  efforts,  we  investigated  research  issues  in  human-computer  interaction, 
knowledge-based  cooperative  systems,  and  explanation  giving. 

4.2.  Previous  Research  Projects  to  Enhance  LlSP-Critic 

Two  previous  research  efforts  in  the  context  of  LlSP-CRmC  provided 
ideas  and  motivation  for  some  of  this  work.  One  effort  investigated  linking  the 
knowledge  contained  in  the  rules  with  a  representation  of  the  user's  knowledge 
using  an  increasingly-complex-microworld  (ICM)  modei  [Fischer  86;  Fischer, 
Lemke,  Nieper-Lemke  88].  Another  developed  on  off-line  statistical  analysis 
component  that  analyzes  the  programmer’s  code. 

Research  surroimding  the  ICM  approach  developed  a  rich  theoretical 
model  which  provides  a  domain  structure  to  guide  users  learning  LISP.  The 
paradigm  accepts  the  claim  from  work  on  learning  environments  that  microworlds 
are  a  powerful  techniques  for  achieving  computer-based  education  [Papert  80].  It 
theorizes  that  one  learns  most  efficiently  when  confined  to  a  subset  of  the  overall 
domain  knowledge  —  a  microworld.  Once  that  microworld  is  mastered  a  learner 
can  progress  to  the  next  more  complex  one  and  continue  to  leam  by  active  ex- 


ploradon,  cridqumg,  access  to  explanations,  and  so  forth.  The  problem  with  this 
approach  lies  in  defining  the  microworlds  for  a  given  domain.  The  idea  is  entic¬ 
ing,  and  a  layered,  onion-like  model  of  the  domain  is  conceptually  neat.  However, 
further  investigations  found  that  perti^s  the  microworlds  were  user,  and  not 
domain,  specific  [Fischer,  Lemke,  Nieper-Lemke  88].  As  individuals,  our 
knowledge  about  any  one  domain  probably  conforms  better  to  a  model  that  looks 
like  a  head  of  iceberg  lettuce,  we  each  learn  a  domain  according  to  idiosyncratic 
microworlds  rather  than  a  canonical  set  of  them. 

The  work  in  this  dissertation  first  considered  user  modelling  in 
LiSP-CRmc  based  on  series  of  microworlds  representing  the  domain.  The  fun¬ 
damental  difficulty  with  that  approach  is  developing  the  underlying  microworld 
structure  for  the  domain  —  a  domain  model  on  which  to  base  the  user  model.  As 
will  be  explained,  it  turned  out  that  a  more  straight-forward  approach  was  a 
concept-based  domain  model. 

The  idea  behind  the  statistical  analyzer  [Fischer  87b]  is  to  process 
programs  written  by  a  user  before  they  are  transformed  by  the  system.  The 
analyzer  collects  data  on  structure  and  use  of  program  constructs.  Such  infor¬ 
mation  as  average  nesting  depth  for  the  hmctions  a  user  defines,  or  the  use  of  cer¬ 
tain  types  of  standard  fimctions  (e.g.,  mapping  or  loop  constructs),  could  provide 
evidence  about  the  expertise  level  of  the  user.  The  idea  here  is  intuitively  attrac¬ 
tive  and  could  be  allied  to  a  broad  range  of  applications.  What  is  required  for  the 
approach  to  be  useful  is  a  set  of  inference  methods  triggered  by  specific  statistical 
data  that  can  classify  a  user  by  expertise  level,  such  as  novice,  intermediate,  or 
expert,  or  into  a  stereotype.  To  determine  these  methods  requires  a  significant 
data-collection  and  analysis  effort  on  a  large  population  of  users,  and  the  correla¬ 
tion  of  those  results  with  an  a  priori  classification  that  is  based  on  an  accepted 
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measurement  instrument,  like  a  test  or  questionnaire.  As  a  technique  for  acquiring 
information  about  users,  statistical  methods  are  important  and  form  a  category  of 
acquisition  methods  in  the  framework  presented  in  Chapter  3.  For  the  time  being, 
the  emphasis  has  been  placed  on  dialog  analysis,  and  the  indirect  implicit  acquisi¬ 
tion  methods;  the  statistical  approaches  were  not  further  investigated. 

The  increasingly  complex  microworld  research  indicated  that  there  is  a 
need  for  a  model  of  the  domain,  one  that  can  provide  a  foundation  for  a  user  model 
representation.  The  statistical  analysis  work  indicatea  ^he  possibility  for  evidence- 
based  user  model  acquisition,  that  evidence  being  the  contents  of  users’  work. 
Evidence  acquired  about  the  knowledge  of  an  individual  programmer  can  then  be 
used  to  assign  them  to  a  specific  expertise  classification.  The  problem  here  is 
similar  to  the  difficulty  with  stereotypes:  both  ideas  have  merit  as  methodologies 
for  acquiring  user  models,  but  depend  on  prior  knowledge  about  the  user  popula¬ 
tion,  knowledge  that  is  not  available  without  a  significant  analytical  effort  The 
^rproaches  investigated  and  implemented  in  this  dissertation  are,  in  some  ways, 
simpler  than  either  of  these  efforts;  we  foimd  that  a  semantic  network  type  concep¬ 
tual  domain  model  of  LISP  can  support  user  model  representation  and  some  in¬ 
direct  acquisition  techniques  (see  Chapter  7).  Next  is  a  description  of  the  architec¬ 
ture  of  the  current  LiSP-CRmC  which  evolved  from  work  on  the  previous  versions. 
This  version  incorporates  the  domain  and  user  models  described  in  this  disser¬ 
tation. 

4  Description  of  Current  Version 

The  current  LISP-CRITIC  system  allows  interaction  between  the  system 
and  the  user  at  the  level  of  individual  transformation  rather  than  entire  files  of 
code;  it  is  being  enhanced  to  provide  context-specific  tailored  explanations  upon 
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request,  and  to  support  some  adaptability  by  users.  The  objective,  to  investigate 
user  modelling,  focuses  on  support  for  explanation-giving.  Instead  of  transform¬ 
ing  an  entire  LISP  program,  handing  it  back  to  the  user,  and  trying  to  explain  the 
differences,  the  design  for  this  version  is  based  on  an  assumption  that  to  achieve  a 
more  coUaboradve  style,  users  should  be  able  to  decide,  on  a  transfonnation-by- 
transformation  basis,  whether  or  not  they  want  each  portion  of  code  changed.  Ftur- 
thermore  the  system  has  to  be  able  to  change  the  code  the  user  actually  wants 
modified,  while  leaving  the  rest  of  the  program  intact;  the  resulting  program  must 
still  compile  and  execute  properly.  Users  need  contextual  access  to  explanations 
of  any  single  suggestion.  These  goals  led  to  the  development  of  a  version  that 
enhances  an  existing,  commonly  used,  program  development  environment,  the 
Symbolics  ZMACS  editor.  Users  can  access  the  critic  at  any  time  while  they  are 
editing  LISP  code  in  23vlACS  (see  Figure  4-7).  The  critic  examines  the  code  and 
makes  one  suggestion  at  a  time;  the  programmer  can  accept  the  recommendation, 
reject  it,  or  request  an  explanation.  When  a  transformation  is  accepted  the  system 
changes  the  code  in  the  editing  buffer. 

A  general  overview  of  the  system  architecture  is  shown  in  Figure  4-4. 
The  user’s  code  is  analyzed  at  what  is  essentially  the  s-expression  level.  When  an 
opportunity  is  found  to  improve  that  expression,  the  systems  produces  an  im¬ 
proved  (optimized)  version  and,  when  the  user  requests  it,  an  explanation.  Inside 
of  LlSP-CRmC  are  a  set  of  engines  and  a  set  of  knowledge-based  components  that 
support  this  process.  This  architectural  diagram  does  not  capture  the  interaction 
between  the  user  and  the  system  that  takes  place;  Figure  6-3  shows  the  interaction 
between  the  system  and  the  user  at  the  process  level. 

Figure  4-5  shows  the  internal  components  in  greater  detail,  and  the  fiow 
of  information  between  them.  Work  on  explanation-giving,  as  instantiated  in  the 
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This  figure  shows  the  architectural  components  of  LisP-CrttiC  and  the 
general  flow  of  data. 

Figure  4-4:  The  Architecture  of  LISP-CRTTIC 


explanation  generator,  is  ongoing  [Fischer,  Mastaglio,  Reeves,  Rieman  90].  As 
discussed  earlier,  the  statistical  analyzer  was  developed  previously  but  has  not 
been  integrated  into  the  system.  The  critic  rules  and  critiquing  component  are 
derivatives  of  work  on  the  initial  versions  of  the  system,  and  have  been  ad^ted  to 
the  Symbolics  environment.  To  provide  a  better  understanding  of  how 
LiSP-CRTnc  operates,  and  the  role  played  by  each  system  component,  an  example 
interaction  will  be  described.  Portions  of  it  will  be  used  in  other  chapters. 

4.4.  Scenario 

In  this  scenario  a  user  is  interacting  with  LiSP-CRTnC.  The  internal  ac¬ 
tions  taken  to  support  the  user’s  decision  process  and  those  performed  by  the  user 
modelling  component  are  not  explained  in  detail  here  but  are  covered  in  Chapter  6 
and  Chapter  7,  respectively.  The  LISP  code  in  this  scenario  was  written  by  an 
undergraduate  Computer  Science  student  enrolled  in  an  introductory  artificial  in- 
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This  figure  shows  the  internal  components  of  LlSP-CRmc  and  the  infor¬ 
mation  flow  between  them. 

Figure  4-5:  Internal  Components  of  USP-CRmc 


telligence  course^  and  comes  from  the  coipus  of  programs  used  in  the  evaluation 
of  the  the  user  modelling  component  described  in  Chapter  8.  It  is  the  program 
developed  for  the  student’s  first  assignment  in  LISP.  An  initial  user  model  (its 
partial  contents  can  be  seen  in  Figure  7-2)  was  provided  to  the  system.  Theoreti¬ 
cally,  the  contents  of  this  initial  model  can  come  fix>m  a  number  of  sources: 

•  Computer-based  tutoring 

•  Explicit  acquisition  approaches,  for  instance  the  use  of  a  questionnaire 

•  Testing  of  the  user’s  knowledge  level 

•  From  a  list  of  concepts  taught  daring  classroom  instruction. 


^This  student,  whose  identity  is  not  revealed,  happens  to  be  male;  therefore,  in 
this  discussion  he  will  be  referred  to  using  male  pronouns. 
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LiSP-CRmc  was  not  used  previously  by  this  student  prograinmer,  there¬ 
fore  the  startup  user  model  here  is  based  on  responses  to  a  data  collection  ques¬ 
tionnaire  completed  by  him,  augmented  with  information  about  concepts  explained 
in  class.  The  theoretical  investigations  of  user  modelling  in  this  research  con¬ 
centrated  on  methods  for  enhancing  an  existing  model  using  the  context  of  the 
user-system  dialog  while  assuming  the  existence  of  some  sort  of  initial  or  start-up 
model  of  each  user.  The  rationale  for  this  assumption  is  the  existence  of  several 
available  techniques  for  providing  the  initial  model  (interactive  questioning  of 
users,  stereotyping,  classification  categories,  etc)  that  could  be  adapted  for  use  in 
LiSP-CRmc.  It  was  felt  that  rather  than  attempting  to  implement  the  entire  range 
of  methods  that  first  build  a  model  "fiom  scratch!*  and  then  improve  it  over  time, 
that  the  work  should  concentrate  on  the  more  difficult  and  less  well  understood 
problem  of  how  to  enhance  that  model  over  time  (dynamically). 

In  the  scenario  the  term  dialog  is  used  to  mean  the  entiie  context  of  the 
human-system  interaction.  The  dialog  notion,  as  applied  here,  will  be  discussed 
mote  extensively  later,  for  this  scenario  it  should  b&  understood  to  encompass  that 
series  of  actions  taken  by  either  a  user  or  a  system  which  the  other  knows  about. 

4.4.1.  First  Dialog  Episode 

The  initial  screen  image  of  the  user  working  on  his  code  in  ZMACS  is 
shown  in  Figure  4-6.  He  wrote  and  debugged  a  program  using  the  ZMACS  editor 
on  a  Symbolics  LiSP  Machine.  From  the  editor  he  invokes  LlSP-C!Rrnc  using  a 
HYPER-S  key  combination.  LlSP-CRUIC  examines  a  single  function  definition 
(defim)  at  a  time.  That  function  definition  is  identified  by  the  system  as  the  one 
within  which  the  user  has  positioned  the  cursor.  For  first  scenario  episode  it  is  the 
defun  for  getop.  The  figure  shows  the  entire  buffer  to  emphasize  that  LlSP-CRmC 
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recognizes  the  user’s  context  (from  the  cursor),  just  as  a  knowledgeable  human 
assistant  might;  the  programmer  does  not  have  to  scroll  the  window  to  a  particular 


configuration  or  mark  a  section  of  the  program  to  identify  it  to  the  critic. 


□ili  -I-  node]  LISPj  SyntM:  Cennon-liepi  I0|  Low«rcM: 


;rh4s  function  will  take,  ar^uneneo  two  Hats  aaeh  canoaad  af  two  aunbara 
}  aaaw»ed  to  be  the  pointa  Xt.Yl  ena  X2«f2. 

lit  will  than  find  the  diatanoa  between  thaaa  two  polnte  waeinf  the  Cuclldaan 
2  diatanca  function  fliwen  to  ua  on  our  aaalannant. 

2  the  soumk  »oot  or  ((xi-xa)"2  *  (Yt-Y2ra) 

(defun  distance  (oone  otwe) 

(aqpt  <*  <*  ('  (ear  oone)  (ear  ptwo))  (-  (car  pone)  (car  ptwo))) 

<*  (-  (cadr  pone)  (eaor  ptue>)  (•  (ea«^  oone)  (codr  ptwo)))))) 


2  The  1  ahere  function  tokca  a  avnbol  and  a  list  of  avnbols  and  their  oppoaltea 
j  It  will  search  for  the  avnbol  In  the  second  Hat.  If  found*  It  will  return 
2  a  Hat  of  80fH  the  word  and  it's  oopoaita. 

(defun  Ishere  (word  ootsble) 

(eond  ((null  oecabla)  nil) 

Knenbar  word  (ear  optaole))  (car  eptaole)) 

(t  (lohero  word  (edr  optabla))))) 


jl  The  function  uses  the  ISHCRC  function  to  to  firet  locate  tha  word 

2  and  1t*a  opposite  m  the  table  of  oppoattea.  than  it  raturna  the  opposite 
2  of  the  orl9lna1  word. 


(defun  sstoo  (word  optoblo) 

E^ono  ((epusi  word  (car  (tahara  word  optabla))) 

I  (cadr  (lanere  word  optsble))) 

(t  (car  (lanere  word  eptaole))))) 


>2  (ho  eaat  function  testa  tna  two  atrin^a  aaainat  each  othar  in  the  fallowing 

!;  1.  If  the  car  of  the  PflTTERh  Hat  ia  a  than  It  la  asaunad  to  be  a 
warleble.  Tha  proarsn  nowea  on  to  the  rest  of  the  tuo  Hats. 

1;  2«  If  the  ear  of  tha  PATTCfttl  Hat  ia  an  aton  than  the  car  of  the  HATCHLIST 
2  It  chockad  to  aaa  if  thav  are  tna  sane.  IC  this  is  true  then  th» 

2  orogran  nowea  on  to  the  root  of  both  1  labs* 

1;  If  neither  of  those  rules  can  be  aatisflad  then  the  function  returns  NIL 
*2  It  returns  t  otherwise 


«(dafun  test  (pattern  natchllst) 

!  (cond  ((and  (null  pattem)(nuH  n«tchHet))t) 

I  ((or  (aoual  (car  oattarn)(car  natehHst)) 

I  (ouestteat  (ear  pattcm>))(teot  (edr  pattem)(edr  notehHst))) 

(t  nil)?) 
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User  Imut 


User  editing  LISP  code  in  the  ZMACS  buffer. 
Figure  4-6:  Scenario-User’s  LISP  Program 


LlSP-CTRinc  examines  the  user’s  code  for  possible  ways  to  simplify  it. 
In  this  case  it  finds  that  a  cond  special  form  could  be  replaced  by  an  j/ special  form 
and  makes  that  suggestion,  as  shown  in  Figure  4-7.  The  user  has  the  choices  in  the 
menu  bar  at  the  bottom  of  the  LlSP-C?Rrnc  window,  of  interest  here  are  the  options 
to  accept,  or  reject  USP-CRmc’s  suggestion,  or  to  ask  the  system  to  explain  this. 
The  user  does  not  understand  the  suggestion,  so  he  selects  the  explain  this  menu 
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option.  The  system  calls  the  explanation  component  which  obtains  from  the 
domain  model  the  concepts  required  to  understand  this  rule.  Then  the  explanation 
component  calls  the  user  modelling  component  to  determine  which  aspects  of  that 
knowledge  the  user  lacks. 


riili  Wo0«tH^P| 


i;Thf«  ul11  take,  as  arguMncdl 

•j  aaatxad  to  b*  the  oointa  XI, Vt  a 
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■;  diatanea  function  a^wen  to  ua  on  own 
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'(defwn  Ishere  (word  ootable) 

I  (eond  ((null  oeteblalf  ntU 
•  ((nanber  word  (ear  optaole))  ( 

I  (t  (ishora  word  (edr  opcabte}) 

» 
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|j  The  tetep  function  uses  the  ISHCRC  f| 
I]  and  It's  eppoaita  in  the  table  of 
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J(dafun  getoe  (word  ootaole) 


Libp-CRITIC 


ftulet  COKtHTO-ir-ELSe  •uleseb;  standard 

(cond  ((«Que)  word  (oar  (ishere  word  opiabic)))  (cadr  (Ishere  word  optablc))) 
(t  (car  (lehere  wei^  oetable)))) 

aas> 

(if  (epuel  word  (ear  (ishere  word  optabte))) 

(cs^  (Ishere  word  opteble)) 

(cer  (Ishere  word  optable))) 


Accept 

Reject 


Lxoiam  This 

Shiow  Current  Function 


Set  Parameui^ 
Cheek  Rules  Statue 


(eond  ((adwal  word  (ear  (itnara  word  optabla))) 

(cadr  (ishere  word  optsbie))) 
<t  (cer  dahere  word  optebleM))) 
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LlSP-C)Rmc  is  accessed  and  suggests  how  to  improve  the  user’s  code. 
Figure  4-7 :  Scenario-User  Invokes  LiSP-CiRmc  on  Function  getop 


The  domain  model  begins  with  the  cond-to-if-else  rule  and,  using  the 
links  between  domain  model  entities,  accumulates  a  concept  set  which  consists  of 
all  prerequisites  to  understanding  the  rule.  The  user  modelling  component  filters 
the  concept  set  so  the  explanation  component  can  focus  on  explaining  only  those 
concepts  which  users  do  not  know.  The  final  step  in  the  explanation  process  is 
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presentation  of  this  information.  The  current  implementation  does  not  contain  a 
fully  developed  presentation  strategy  so  it  uses  a  simple  strategy  of  choosing  to 
explain  the  first  three  concepts  in  the  filtered  list,  in  this  case  predicates,  con¬ 
ditionals,  and  tests  and  displaying  hypertext  explanations  for  them.  To  create  a 
more  realistic  scenario,  mock  e:(planations  using  these  three  concepts  as  a  basis 
are  displayed  in  Figure  4-8.  These  are  more  in  line  with  what  we  would  expect  a 
fully  competent  explanation  strategy  to  produce.  In  the  present  system  a  followup 
capability  is  provided  for  with  hypertext.  Clicking  on  any  of  the  terms  shown  in 
bold  causes  an  explanation  associated  with  that  object  in  the  domain  model  or  a 
description  from  the  Symbolics  Document  Examiner’s  documentation  to  be  dis¬ 
played.  The  explanation  in  Figure  4-8  provides  accesa  to  explanations  of 
s-expressions,  tests,  cond,  if,  nil,  and  non-nil. 


USP-CRITIC  explains  a  suggestion  based  on  the  cond-to-if  rule  to  include 
those  prerequisite  concepts  the  user  does  not  know. 

Figure  4-8:  Explanation  For  cond-to-if-else  Rule 
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The  user  accepts  this  suggestion,  and  LiSP-CRmc  automatically  rewrites 
the  modified  portion  of  the  user’s  code  in  the  editing  bufier.  In  Figure  4-9  the 
body  of  function  getop  has  been  changed  to  reflect  the  cond-to-if-else  transfor¬ 
mation.  In  this  function  definition,  LISP-Crttic  found  only  one  transformation  to 
suggest,  therefore  at  this  point  the  programmer  is  returned  to  his  code  editing  buff¬ 
er  and  LiSP-CRmc’s  window  becomes  inactive;  it  moves  into  the  background,  out 
of  the  programmer’s  view. 
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After  the  user  accepts  USP-CRmc’s  suggestions,  the  system  modifies  the 
code  in  his  bufier. 


Figure  4-9:  Modified  ZMAC5  Bufier 


The  user’s  actions  throughout  this  episode  trigger  changes  in  his  user 
model.  Those  specific  changes  are  described  in  detail  together  with  an  explanation 
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of  the  user  modelling  process  that  precipitate  them  in  Chapter  7.  In  general,  any 
action  taken  by  the  user,  or  information  received  by  him  in  the  form  of  explana¬ 
tions,  trigger  those  changes.  The  user’s  recent  of  an  explanation  of  the  cond-to-if 
rule  and  concepts  behind  it,  and  the  fact  that  he  made  a  decision  to  accept  the 
transformation  trigger  changes  to  his  user  model  (that  updated  model  is  partially 
displayed  in  Figure  7-4).  These  direct  changes,  in  turn,  uigger  indirect  inferences. 

4.4.2.  Second  Dialog  Episode 

The  second  scenario  episode  could  occur  immediately,  or  at  a  later  time; 
the  user  model  is  saved  between  sessions  and  reused  when  the  user  subsequently 
accesses  LlSP-CRmc.  Our  user  now  requests  LlSP-CRmC  to  look  over  the  code 
written  for  function  test  which  causes  a  recommendation  based  on  a  rule 
de-morgan  (the  rule  applies  DeMorgan’s  law  from  logic  to  combine  booleaits)  and 
again  he  requests  an  explanation.  The  explanation  component  once  again  consults 
the  domain  and  user  models  to  detezmina  what  needs  to  bo  explained  to  this  par¬ 
ticular  user.  Those  top  three  concepts  selected  by  the  simplified  explanation 
strategy  are  logical  functions,  internal  representation,  and  arguments;  these  are 
integrated  into  the  complete  text  of  the  mock  explanation  shown  in  Figure  4-10. 

The  user  accepts  the  suggestion  and  the  system  shows  the  user  a  second 
suggested  transformation  for  this  piece  of  code;  it  is  based  on  the 
cond-erase-pred.t  rule,  and  the  user  asks  the  system  to  explain  it;  Figure  4-11 
shows  that  explanation.  Our  scenario  user  also  accepts  this  suggestion.  A  third 
rule,  cond-erase-t.nil,  triggers;  it  is  very  similar  to  the  previous  rule  therefore,  the 
user  accepts  USP-CRmC’s  recommendation  without  explanation.  He  is  able  to 
generate  his  own  explanation  because  he  knows  a  similar  rule  that  was  previously 
encountered,  and  is  familiar  with  the  underlying  concepts.  Throughout  the  dialog. 
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The  function  tooto  tho  two  atringg  osatnot  oodi  otltor  In  tno  following 

uoyt 

1.  Xf  the  cor  of  the  PATTERtl  Hat  la  o  *?*  than  it  la  ooawno4  to  bo  o 
worlobic.  Tho  orogron  nowco  on  to  the  neat  of  tho  two  Hato. 
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Xf  nolthor  of  thoao  ruloa  can  bo  aarliflod  thon  tho  function  rtturna  hXL 
It  retumo  1  othcrwlac 


dofun  toot  (oattom  natchtiat) 

(eond  ((and  (null  oattornKnuH  natchllat))!) 

I  ((or  (aquol  (ear  oottomXear  natchllat)) 

(ouoatteat  (car  oattern>}}(C«at  (edr  oatbomXcdr  natchllat))) 

(t  mun  _ 

I  Li«p-CRITl6 


Tho  nacehup  function  aaaun^j 
TEST  function. 

Xt  thon  ulll I 

1.  Xf  tho  ear  of  tho 

of  tho  nATCHLXST  then 
nowoa  on  to  the  root  e^| 
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(defun  natchup 
(eond 
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((null  oatta^l 
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Thg  function  natch  will  f1i 
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function  to  eroata  a  Hat 
Xf  tho  ruloa  aro  not  not. 
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(null  natehliat)) 

«•«> 

(not  (or  oattom  natchllat)) 
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In  logical  aynoelat 

pttttom)  ^  mokehllot)  ■  ^(pattom  •*  matchlldt) 

Logical  funetlona  In  LXSf  oorfem  booloan  eparatlona.  Tho  argunonta 
ara  aynboU  raortaantad  aa  all  (falaa)  or  thay  hawo  a  walwa  and  ara 
therefero  true.  AH  aynbala  and  values  are  captured  Internally  in  a 
rceraaantation  aehana,  for  ananola  Hats  ara  eoHoetlana  af  caoa  ctlla. 
float  LISA  functions  taka  argunehts.  a  function  doflnod  aa 
(dofun  foo  (x  y)  ...) 

has  argunanta  n  and  y  that  ara  given  values  uhan  foo  la  eallad  and 
then  used  m  the  function  body. 
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'(defun  natch  (oattern  natchllat) 

(eond  ((  taat  oattorn  natchllatlCnatehuo  pattern  natchllat)) 
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CTua  S  Jun  4t3fil7]  Kayboard  CL  US€#J  Ifaar  Input 

LiSP-diRmc  is  asked  to  examine  another  part  of  the  user’s  program,  a  sug¬ 
gested  transformation  and  its  explanation  is  shown. 

Figure  4- 10:  Explanation  For  de-morgan  Rule 

the  user  model  is  updated  each  time  the  user  receives  an  explanation  or  makes  a 
decision. 


4.4.3.  Third  Dialog  Episode 

In  a  third  dialog  episode  our  programmer  asks  LlSP-ITRmc  to  examine 
his  definition  for  the  function  match.  The  systems  reconunends  that  an  if  he  used 
in  place  of  a  eond  (see  Figure  4-12).  This  transformation  is  based  on  the  same 
cond-to-if-else  rule  that  fired  in  the  first  episode,  and  because  the  user  already  en¬ 
countered  this  same  rule  and  had  it  explained,  he  accepts  the  suggestion  without 
requesting  explanation.  LlSP-(ZRmc  changes  the  user’s  program  code.  The  final 
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Lisp-CRITIC 


Rule:  COHD-ERRSE-PRED.T  Ruleset:  standard 

(cond  uynot  (or  pattern  natcKliat))  t)l 

(T^r  (equal  (car  pattern)  (car  natch list)) 

(questtest  (car  pattern))) 

(test  (cdr  pattern)  (cdr  natch list))) 

(t  nil)) 

SSS> 

(cond  ((not  (or  pattern  natcniist))) 

((or  (equal  (car  pattern)  (car  natch! ist)) 

(questtest  (car  pattern))) 

(test  (cdr  pattern)  (cdr  natch! ist))) 

(t  nil)) 

You  have  specified  the  syabal  t  as  the  return  value  for  one  clause 
in  a  cond.  This  creates  entra  code  that  reduces  readability.  It  is  not 
required  because  the  value  of  the  test,  a  lisp  atee,  will  be  returned 
when  the  test  is  true.  Rn  aton  in  LISP  is  either  the  ear  or  cdr  of  a 
cens'CcM.  It  can  be  a  syabol  representing  a  variable  or  value,  or  a 
nunber.  Rny  expression  that  does  not  evaluate  to  nil  is  considered  to 
be  true,  hil  and  the  enpty  list  ()  are  equivalent  and  in  testing 
functions  considered  to  be  false. 


Lxplatn  this 

Show  Current  Function 


Set  Parameters 
Check  Rules  Status 


LiSP-CRmC  recommends  a  second  transfoimation  in  the  defun  for  test, 
which  the  user  asks  to  be  explained. 

Figure  4-11:  Explanation  For  cond-erase-pred.t  Rule 


state  of  his  editing  buffer  with  is  shown  in  Figure  4-13,  the  contents  of  his  user 
model  is  partially  shown  in  Figure  7-6  and  its  completCL  internal  representation  in 
Appendix  B.  That  model  has  changed  during  the  scenario  and  if  another  explana¬ 
tion  of  the  cond- to- if  rule  had,  in  fact,  been  requested,  the  user  modelling  com¬ 
ponent  can  provide  to  the  explanation  component  the  fact  that  the  rule  was 
previously  explained  together  with  a  new  concept-set  to  use  for  presenting  the  ex¬ 
planation  this  time. 

The  development  of  the  explanation  conqx)nent  has  not  progressed  past 
the  conceptual  and  methodological  stages.  The  explanations  shown  in  this 
scenario  are  only  intended  to  point  out  the  relationship  between  the  work  on  the 
domain  and  user  modelling  undertaken  in  this  dissertation,  and  the  requirements 
for  explanation-giving.  As  presented,  the  explanations  do  not  constitute  a  finished 
product  and  should  be  used  primarily  as  a  vehicle  to  tmderstand  how  the  system 
makes  choices  of  what  to  present  during  its  dialogs  with  the  user.  Additional  work 
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In  the  third  dialog  episode  LiSP-CRmc  examines  code  for  the  defun  match 
in  the  user’s  program. 


Figure  4-12:  Scenario-User  Invokes  LlSP-CRITIC  on  defun  match 


to  implement  the  presentation  strategies  for  explanation-giving  is  required.  This  is 


not  a  trivial  problem;  it  is  one  of  the  three  major  issues  in  explanation  identified  in 


[Chandrasekaran,  Tanner,  Josephson  89];  the  other  two  being  the  system’s  under¬ 


standing  or  deep  model  of  the  domain  and  user  modelling.  Recent  LlSP-CRmc 


work  has  concentrated  on  these  latter  two  problems. 


4  Summary 


Developing  a  marketable  system  is  not  the  ultimate  goal  in  the 
LiSP-CRmc  work,  instead  a  prototyping  process  was  followed;  it  is  designed  to 
help  achieve  a  better  understanding,  at  a  conceptual  level,  of  what  is  required  from 
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(car  (i*h«rc  word  optctoU)))) 


Th«  Cc9t  function  ttats  th«  two  strings  o^inoc  aoch  ochor  in  tho  following 
woy: 

1.  If  tho  cor  of  tho  PATTERtl  Hat  ia  a  tnan  it  io  aaawnad  to  b«  a 
uariablt.  li>o  progran  nowaa  on  to  tna  root  of  tho  two  Hoto. 

3.  If  tha  cor  of  tho  PflfTERtI  Hot  ia  an  aton  than  tho  car  of  tho  MTCMLIST 
la  chockod  to  aoa  if  thoy  aro  tho  aana.  If  thia  io  trwo  than  tho 
progran  novea  on  to  tho  rcat  of  tMth  Hsta. 

If  naithop  of  thoaa  rulaa  can  ho  aatiafiad  than  tho  function  roturno  NIL 
It  potuma  I  othorulao 


dafun  toat  (pattorn  natchllat) 

(cond  ((not  (or  pattern  natchHat))) 

((or  (equal  (ear  pattorn)  (car  natehliot)) 
(quoattoat  (ear  pattern})) 

(teat  (edr  pattern)  (cdr  natehliot))))) 


Tho  natehup  function  aaaunca  that  the  two  Hate  are  logoi  aa  dafinod  by  the 

TEST  function. 

It  then  ui 1 1 1 

1.  If  the  ear  of  tha  PflTTERH  Hat  la  an  aton  and  It  ia  tho  aano  aa  the  cw* 
of  the  nRTCM.IST  then  it  netchea  and  is  thrtMt  away.  The  progran 
nowoo  on  to  tho  peat  of  tha  Hat. 

2.  It  uiH  now  nafca  a  Hat  out  of  tho  cdr  of  tho  car  of  the  PfITIERII  Hat  and 
tho  ear  of  the  nOTCMLiSf.  (Tho  wariable  and  the  natch)  It  will  then  nowe 
on  to  tho  rcat  of  the  Hat. 


(defun  natehup  (pattern  natehliot) 

(cond  ((null  pattern)  nil) 

((eoual  (ear  oattarn)  (ear  natehliot))  (natcnup  (edr  pattern)  (edr  natchHat)!) 
(t  (cona  (append  (cdar  pattern)  (car  natchHat)) 

(natehup  (edr  pattern)  (edr  natchHat)))))) 


The  function  natch  will  first  uaa  the  ^6T  function  to  ace  if  the  two 
Hata  neat  tha  requirenanta  nentioned  aeouo.  If  ao»  It  waaa  tha  natehup 
funetion  to  craata  a  Hot  out  of  tna  warlblas  and  thair  apropriata  natehea 
If  tha  rulaa  are  not  nat,  it  returns  NIL. 


‘(dafun  natch  (pattern  natchHat) 

■  (If  (taat  oattarn  natchHat)  (natehup  pattam  natchHat)  ml)) 


ea  (Lli(^^~cooe'^^r  »acenar  io-uaer 
(12U0>99  Proceaa  Screen  Hordcopy  wonta  to  type  out] 


Mo«*,o-L:  liovo  point;  f-iouse-H;  Hark  word;  Mouso-h:  tJitor  monu. 
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(Sixi  11  fed  I2tl9:d4j  Keyboard 


»t>V-  ft  -.nil)) 


Abort . 


Figure  4-13:  Final  State  of  Editing  Buffer 


a  user  model,  and  to  inform  the  process  of  developing  a  general  approach  to  ac¬ 
complishing  that.  A  critic  was  used  as  a  context  for  investigating  ideas  and  im¬ 
plementing  some  of  that  user  modelling  framework  because  the  paradigm  is  well 
understood,  and  has  been  instantiated  in  at  least  one  mature  and  well  understood 
system,  LiSP-dJRmc. 

The  ideas  for  how  to  model  users  of  critiquing  systems  have  their  foun¬ 
dation  in  theoretical  notions  about  human-computer  interaction  and  grew  out  of 
studying  user  modelling  in  other  domains.  Those  ideas  served  to  guide  implemen¬ 
tation  of  the  user  modelling  component  in  LlSP-CRITlC.  It  is  one  of  the 
knowledge-based  components  of  tlK  system,  the  others  are  the  critic  rules  and  the 
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conceptual  domain  model.  A  related  component  is  the  explanation  generator;  it  is 
supported  by  the  capabilities  of  the  user  modelling  component. 

In  the  past,  LlSP-CRmc  has  been  a  platform  for  evaluating  ideas  about 
how  computer  systems  should  be  designed.  Integrating  a  user  modelling  com¬ 
ponent  is  a  natural  extension  of  that  previous  work;  the  story  is  actually  more  com¬ 
plex:  LiSP-CRmC  was  not  merely  extended,  but  ported  to  a  new  computational 
environment  and  adapted  to  a  new  interaction  style  in  support  of  this  research.  In 
the  next  section  of  this  dissertation  (Chapters  5,  6,  and  7)  the  system  con^nents 
which  were  developed  in  and  are  directly  related  to  this  work,  the  domain  model, 
the  envisioned  approach  to  explanation,  and  the  user  modelling  component,  are 
described  in  that  sequence. 


CHAPTER  V 

A  DOMAIN  MODEL  FOR  USP 

5.1.  Introduction 

This  chapter  describes  the  domain  model  developed  to  support 
explanation-giving  and  user  modelling  in  USP-CRinc.  The  work  refines  and 
implements  ideas  developed  in  previous  research  [Fischer  88a].  In  this  chapter,  I 
cover  why  there  is  a  need  for  a  domain  model,  how  the  domain  was  analyzed,  the 
results  which  in  turn  determined  the  model’s  general  form,  and  then  the  graphical 
notation  used  to  conceptualize  the  model.  Next  I  discuss  the  implementation  of 
the  domain  model  followed  by  the  extensions  and  other  potential  uses  for  both  the 
model  and  the  methodology. 

Developing  the  domain  model  was  an  enabling  technology  that  was  re¬ 
quired  in  order  for  both  the  explanation  and  user  modelling  research  to  proceed. 
The  development  of  that  model  and  its  ultimate  form  are  described  here  because 
development  did  involve  significant  effort  and  knowing  how  the  model  was 
developed  and  then  implemented  in  LiSP-CRmC  will  facilitate  the  reader’s  under¬ 
standing  when  I  describe  its  use  in  the  explanation  process  in  Chapter  6,  and  for 
user  model  representation  and  acquisition  in  Chapter  7. 

The  model  represents  the  domain  of  LISP  in  terms  of  three  entities:  the 
concepts  of  the  language,  basic  COMMON  LISP  functions,  and  the  transformation 
rules  in  USP-CRmc.  The  latter  are  represented  in  the  conceptual  domain  mode, 
even  though  they  are  captured  in  applicative  form  in  the  rule  base,  because  a  rule 
is  the  triggering  condition  for  an  explanation. 


5^.  Requirements  for  a  Domain  Model 
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As  previously  discussed,  in  order  to  accomplish  cooperative  problem 
solving  it  is  imperative  that  systems  have  the  c^ability  to  explain  their  reasoning. 
In  the  case  of  critiquing,  in  general,  and  USP-CRmc  in  particular,  this  means  ex¬ 
plaining  the  reason  a  given  transformation  is  being  suggested  —  the  rationale  and 
concepts  behind  a  rule. 

A  common  theme  of  other  research  in  explanation  is  that  in  order  to 
provide  an  acceptable  explanation  capability,  the  system  needs  to  represent 
knowledge  of  the  subject  domain  at  an  abstract  level  [Qancey  87;  Kass,  Finin  88b; 
Paris  87;  Wiener  80];  I  say  more  about  this  in  Chapter  6.  For  USP-CRTTIC,  pre¬ 
vious  research  determined  the  possibility  of  representing  programming  knowledge 
in  terms  of  concepts,  programmer  goals  and  functions  [Fischer  88a],  Such  a 
representation  can  explain  the  improvements  suggested  by  the  rules  and  derive  a 
model  of  the  user.  The  implementation  of  the  domain  model  described  here  sup¬ 
ports  those  goals. 

The  rule  base  in  LlSP-CRmc  represents  procedural  knowledge  in  a  com¬ 
piled  or  applicable  form  that  is  appropriate  for  efficiently  analyzing  code  and 
rq}idly  generating  recommendations  for  how  to  improve  segments  of  that  code. 
However,  knowledge  in  this  fonn  will  only  support  rule  tracing  explanation  ap¬ 
proaches  [Scott,  Qancey,  Davis,  Shortliffe  84]  and  these  were  shown  to  be  inade¬ 
quate  [Qancey  84];  systems  need  the  ability  to  explain  rules  at  the  concept  level  so 
as  to  facilitate  user  understanding  and  support  learning.  To  achieve  that  requires  a 
more  abstract  domain  model;  a  model  that  captures  the  abstractions  representing 
the  underlying  domain  at  the  level  of  its  fundamental  concepts. 

A  conceptual  structuring  of  the  domain  should  provide  a  way  to  link  the 
applicative  rule-base  knowledge  with  explanation  strategies  and  with  the  user 
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model.  In  LlSP-CRmc,  a  rule,  or  set  of  rules,  is  the  underlying  causative 
mechanism  behind  a  single  piece  of  advice.  To  understand  that  advice  well 
enough  to  decide  whether  or  not  to  accept  it,  users  needs  to  understand  what  con¬ 
cepts  underlie  that  rule.  A  concept-based  domain  representation  can  be  configured 
so  that  it  provides  that  information;  it  can  inform  the  system  what  concepts  under¬ 
lying  that  rule.  In  turn,  the  system  needs  to  be  able  to  determine  which  of  these 
concepts  are  not  part  of  the  user’s  current  knowledge  so  it  can  focus  on  explaining 
the  unfamiliar  concepts.  The  terms  understanding  and  knowing  are  used 
synonymously;  we  do  not  try  to  make  the  theoretical  distinctions  between  them 
that  are  important  to  some  studies  of  cognition  or  philosophy. 

5.3.  Form  of  the  LISP  Domain  Model 

In  order  to  support  the  explanation  strategies  and  user  modelling  process 
in  LiSP-CRmc,  the  information  contained  in  the  domain  model  consists  first  of  the 
underlying  concepts  for  LISP.  To  determine  those  concepts  we  reviewed  the  fol¬ 
lowing  commonly  used  USP  texts:  [Steele  84],  [Winston,  Horn  81],  and  [Wilensky 
84].  Forty-five  commonly-refened-to  concepts  were  identified  in  these  texts.  The 
list  does  not  include  more  fundamental  concepts  that  exist  "in  the  world",  such  as 
the  set  of  integers,  but  focuses  on  those  concepts  that  are  unique  to  LISP  or  pro¬ 
gramming  languages  in  general.  The  terms  used  to  identify  these  concepts  are 
listed  alphabetically  in  Figure  5-1.  The  term  concept,  as  expressed  in  the 
Philosophy  of  Science  literature,  is  an  abstraa  notion;  there  is  a  distinction  be¬ 
tween  concepts  themselves  and  the  terms  that  stand  for  them  [Hempel  65].  The 
concepts  shown  in  Figure  5-1  were  designated  using  terms  that  seemed  ap¬ 
propriate,  while  recognizing  that  in  other  research,  and  context,  they  may  be 
described  with  other  names.  In  comparing  this  analysis  with  an  effort  by  Gray  to 
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capture  the  underlying  entities  of  LrSP  in  a  hypertext  database  [Gray  88],  we  found 
sufficient  overlap  in  terms  and  structure  to  provide  confidence  that  the  topology  is 
valid  and  useful.  His  work  could  not  be  used  directly  because  it  was  never  com¬ 
pleted. 


ARGUMENTS 

LISTS 

ASSOeXAnON-USTS 

UTERAUQUOTE 

CAR-CDR-OONCATENATION 

LOOICAl^FUNCnONS 

cxxrorncM^Ai^Exrrs 

MAPPING 

OMmmewALs 

MULTI-VALUE-RETURN 

OONS-CEU. 

NUMERIC-ITERATION 

DATA-TYPES 

OPTHWAL-PARAMETERS 

DESTRUCnVE-FUNCnONS 

OUTPUT-FUNCnONS 

DOTTED-PAIR 

PARALLEL/SEQUENTIAL-BINDING 

EMBEDDED-FUNCnC^S 

PARAMETERS 

EVALUATION 

PREDICATES 

EVALUATION-ORDER 

PROPERTY-L^S 

FALSE/EMPTY-UST/NIL 

RECURSION  ' 

FUNCnON-DEFINmON 

SCOPE 

PUNCnONS 

SIDE-EFFECTS 

IDENTmr-VS-BQUIVAlJENCE 

STRINGS 

INPUT-FUNCTIONS 

SYMBOUC-EXPRESSION 

INTERNAL-REPRESENTATION 

TAIL 

riERATKW 

TESTS 

LAMBDA-BINDINO 

TRUE/NON-NIL 

USP-ATOM 

VARIABLE-INTTIALIZATION 

UST-riERATION 

VARIABLES 

Figure  5-1:  List  of  Domain  Concepts 


While  selecting  the  set  of  concepts  for  inclusion  in  the  domain  model, 
two  types  of  relationships  between  concepts  were  recognized,  relationships  that 
are  useful  for  explanation-giving,  and  one  that  can  be  used  in  user  model  acquisi¬ 
tion: 

1.  The  dependent-on  relationship:  This  indicates  for  a  particular  con¬ 
cept  which  other,  more  fundamental,  concepts  are  prerequisites  to 
understanding  it. 
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2.  The  related-concepts  relationship:  This  is  a  relationship  between 
concepts  that  are  similar,  this  information  could  be  used  by  the  ex¬ 
planation  component  in  some  presentation  strategies. 

We  also  selected  103  fundamental  LISP  functions  to  be  represented  in 
the  domain  model,  primarily  those  that  are  found  in  the  LlSP-CRITIC  rules.  The 
term  “function”  is  not  entirely  correct,  this  class  of  domain  entity  might  more 
specifically  be  referred  to  using  the  term  “constructs”,  as  in  [Steele  84].^ 
However,  “Function”  is  the  term  used  here  because  the  it  was  selected  at  the 
beginning  of  the  domain  analysis  and  continued  to  be  used  throughout  the  im¬ 
plementation.  To  understand  a  function  also  depends  on  understanding  certain  un¬ 
derlying  concepts;  therefore,  functions  are  related  to  concepts  via  "dependent-on" 
relationships,  like  the  one  described  above.  Functions  may  also  be  similar  to  one 
another,  for  example,  cond  is  similar  to  if.  the  model  also  captures  this  relation¬ 
ship. 

Part  of  the  analysis  process  was  a  grouping  of  the  concepts  and  functions 
into  logically  related  sets  by  several  LISP  programmers.  We  followed  a  methodol¬ 
ogy  that  has  been  successfully  used  to  stmeture  similar  domains  in  other  research 
[Doane,  Pellegrino,  Klatsky  89].  The  45  concepts  were  divided  into  five  groups 
that  represent  an  approximate  consensus  of  the  experts’  categorizations.  The 
groups  seem  to  fit  into  a  hierarchy  when  viewed  across  the  "dependent-on"  layer 
of  relationship,  but  this  attribute  was  not  further  investigated.  These  groupings  are 
shown  in  Figure  5-2;  the  names  assigned  attempt  to  imply  the  commonality  that 
exists  between  the  concepts  in  that  group.  Similarly  the  103  functions  were  classi¬ 
fied  into  14  categories.  The  rationale  for  the  categorization  exercise  was  to 

^Still  more  precisely,  the  set  actually  consists  of  special  forms  and  standard 
macros  defined  for  COMMON  LISP. 


validate  the  domain  entities  that  had  been  selected,  and  to  integrate  the  knowledge 
of  other  domain  experts  into  our  specification  for  LISP.  This  exercise  was  also  a 
way  to  reflect  on  and  refine  the  concepts.  The  purposes  for  which  these  groupings 
might  be  used  in  LiSP-CRmC  are  not  yet  established,  but  that  part  of  the  process  is 
discussed  here  to  demonstrate  the  depth  of  the  analysis  and  the  generality  of  the 
modelling  approach.  The  categorizations  have  been  captured  in  the  domain  model 
for  possible  future  use. 

We  also  capture  LiSP-CRmC  rules  in  the  domain  structure  because  this 
is  the  level  of  qiplicadon  knowledge  used  by  the  system  and  for  sake  of  having  a 
complete  single  representation  of  the  system  knowledge.  A  rule  has  links  to  the 
functions  that  occur  in  the  rule,  and  the  USP  concepts  that  underlies  iL  When  the 
system  recommends  a  change  to  a  program,  the  only  thing  it  knows  is  that  the 
same  code  conformed  to  a  pattern  expressed  on  the  left  hand  side  of  that  rule  and 
that  it  could  be  rewritten  accorduig  to  the  pattern  on  the  right  hand  side.  To  model 
what  is  involved  in  understanding  that  rule,  it  was  necessary  to  capture  knowledge 
about  the  functions  in  the  rule  and  any  domain  concepts  that  are  behind  it.  Con¬ 
cepts  and  fimctions  probably  exist  as  part  of  a  programmer’s  mental  model  of  the 
domain  [Genmer,  Stevens  83],  therefore,  these  parts  of  the  domain  model  may  be 
something  close  to  a  cognitive  representation,  possibly  representing  chunks.  It  is 
unlikely  that  users,  with  a  few  exceptions,  retain  a  LiSP-CRmc  rule  as  part  of  their 
mental  riKKlel  for  the  domain,  even  after  they  develop  an  tmderstanding  of  it. 

In  summary,  the  domain  model  needs  to  capture  three  types  of  entities 
Lisp  concepts.  Lisp  functions,  and  the  USP-CRmc  rules,  together  with  the 
relationships  between  instances  of  them.  Relationships  are  often  one-to-many,  but 
their  topology,  although  somewhat  hierarchical  within  certain  relationships  (like 
the  dependent-on-concepts  for  aU  concepts  in  the  model),  is  highly  interconnected 
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Variable  Initialization 

Numeric  Iteration 

List  Iteration 

Identity  vs  Equivalence 

Figure  5-2;  Grouping  of  Concepts 


and  acyclic.  Several  paradigms,  such  as,  frames  and  semantic  networks  were  con¬ 
sidered  as  possible  representation  schemes  for  the  model. 

5.4.  Conceptual  Graph  Notation  For  Representing  the  Domain  Model 

An  reproach  that  provides  the  ability  to  visualize  the  entities  and  the 
relationsh^s  from  the  analysis  above  was  conceptual  graph  notation;  it  also  helped 


Conceptual  graph  showing  concept  A  is  related  to  concept  B  by  relation 
Rl. 

Figure  5-3:  Conceptual  Graph  Notation 


An  exan^Ie  of  how  this  representation  allows  visualization  of  the 
domain  model  for  LISP  is  in  Figure  5-4;  it  shows  the  LISP  concept  Recursion  using 
this  notation.  In  this  example.  Recursion  is  dependent  upon  the  LiSP  concepts 
Tests,  Conditionals,  and  Functions’,  it  is  related  to  the  concept  of  Iteration.  Recur¬ 
sion  is  not  a  concept  undeilying  any  LlSP-CRTnC  rule,  but  because  it  is  one  com¬ 
monly  used  in  most  LISP  texts,  it  has  been  c^tured  in  the  domain  model.  It  is 
shown  here  to  demonstrate  the  generality  of  the  q}proach.  The  conceptual  domain 
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model  should  be  able  to  serve  purposes  more  general  than  explanation-giving  and 
user  modelling  in  critics.  Using  recursion,  a  domain  concept  not  required  by  our 
system,  demonstrates  generality,  and  should  provide  an  intuition  to  the  reader  for 
how  the  conceptual  graph  model  approach  might  be  applied  to  serve  other 
paradigms  and  applications. 


Figure  5-4:  Exan^le  of  LISP  Concept  Recursion  in  Conceptual  Graph  Notation 


Conceptual  graphs  were  a  useful  methodology  for  visualizing  the 
domain  model,  but  an  implementation  method  was  required.  For  reasons  of  por¬ 
tability,  availability,  and  standardization,  the  domain  model  for  LrSP-CRmc  was 
implemented  in  the  Common  LISP  Objects  Systems  (Clos)  extension  to  COMMON 
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5^.  Implementation  of  the  Domain  Model 

In  the  domain  model  implementation,  the  concepts  (rectangles)  from 
Conceptual  Graph  notation  were  defined  as  classes,  and  relations  between  con¬ 
cepts  (circles),  captured  in  slot  definitions.  The  class  hierarchy  for  LISP  consists  of 
a  super  class,  lisp-object,  with  three  subclasses  lisp-concepts,  lisp-functions  and 
lisp-critic-rules.  There  are  slots  in  each  object  instance  for  name;  dependent-on- 
concepts;  related-concepts,  lelated-fiinctions,  and  related-rules;  and  the  groupings 
shown  in  Figure  5-2.  Group  membership  for  lisp-concepts  is  represented  in  the 
level  slot,  such  as  recursion  belonging  to  the  category  of  intermediate  shown  in 
Figure  5-7,  for  lisp-functions  in  the  category  slot.  The  CLOS  code  that  defines 
these  objects  is  shown  in  Figure  5-5.  The  three  types  of  entities  inherent  common 
slots  for  name  and  dependent-on-concepts  from  the  fundamental  class  lisp-object. 

The  complete  domain  model  is  difficult  to  show  graphically  because  it  is 
highly  interconnected.  It  can  be  considered  to  have  thuree  layers,  one  each  for  USP 
concepts,  functions,  and  LISP-CRTTIC  rules.  Populating  each  layer  are  instances  of 
the  entity  class  for  that  level.  T.inlcs  are  found  between  instances  within  a  level,  as 
well  as  between  instances  at  different  levels.  For  example  the  LiSP-CRmc  rule 
cond-to-if  is  found  in  the  LlSP-CRmc  rule  layer,  it  is  linked  both  to  similar  rules 
(e.g.  cond-to-when)  within  that  layer,  as  well  as  to  concepts  (e.g.,  conditionals)  in 
the  concept  layer,  and  of  course  to  functions  (e.g.,  cond)  in  the  function  layer. 

To  give  the  reader  a  flavor  for  the  interconnectivity  of  the  model,  again 
consider  the  LISP  concept  recursion.  Understanding  recursion  is  dependent  on  the 
user  understanding  the  concepts  of  tests,  conditionals,  and  functions.  Recursion  is 
also  related  to  the  concept  of  iterations.  The  code  to  instantiate  Recursion  as  an 
instance  of  a  lisp-concept  is  shown  in  Figiue  5-7;  the  conceptual  graph  represen¬ 
tation  for  that  concept  in  Figure  5-4.  Most  concepts,  functions,  and  rules  in  the 
domain  model  have  similar  high  degrees  of  connectivity. 
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(def class  LISP-OBJECT () 

; ; ;  Generic  Super  Class  for  all  LISP  Objects 
( (name 

: accessor  name 
: init arg  : name ) 

( dependent -on- concepts 

: accessor  concept s-dependent -on 
:init£orm  nil 

: initarg  : dependent-on-concepts) ) ) 

(defclass  LISP-CONCEPT  (lisp-object) 

( (related-concepts 

: accessor  related-concepts 
: init form  nil 

: initarg  : related-concepts) 

(level 

: accessor  level 
: init form  'high-level 
: initarg  ; level) ) ) 

(defclass  LISP-FUNCTION  (lisp-object) 

( (pattern 

: accessor  syntax-pattern 
: initarg  : pattern) 

(related-functions 

: accessor  related-functions 
: init form  nil 

: initarg  : related-functions) 

(category 

: accessor  category 
:initform  'unclassified 
: initarg  : category) ) ) 

(defclass  LISP-CRITIC-RULE  (lisp-object) 

( (functions -in- rule 
: accessor  functions 
:  initarg  :functions-in-rxile) 
(related-rules 

: accessor  related-rules 
:initform  nil 
; initarg  : related-rules 
))) 


Figure  5-5:  Clos  Specification  For  USP  Domain  Entities 
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It  is  difficult  to  display  die  entire  domain  structure  in  a  single  two 
dimensional  graph.  In  Figure  5-6  we  provide  a  feel  for  the  complexity  of  the 
structure  when  viewed  across  a  portion  of  a  single  strata  or  level  of  the  domain.  It 
shows  graphically  the  lisp-concept  layer  together  with  the  dependent-on-concepts 
links  between  the  45  concepts.  For  simplicity  and  readability  sake,  this  figure 
does  not  use  the  conceptual  gn^h  notation;  instead  each  oval  represents  a  LISP 
concept,  and  links  are  all  of  the  same  type;  they  represent  the 
dependent-on-concept  relation.  Different  oval  sizes  represent  each  of  the  5 
categories  shown  in  Figure  5-2. 

The  primary  reason  an  abstract  domain  representation  for  LISP  was  in¬ 
vestigated  was  the  need  to  support  explanation-giving  and  user  modelling.  In  tfiat 
regard  the  the  model  can  support  bodi  of  these  processes  in  several  ways. 

The  explanation  compoi»nt  uses  die  domain  model  to  detemaine  what 
concepts  must  be  explained  to  a.user  whadoes  not  understand  a  paiticulai:  recom¬ 
mendation.  Since  all  recommeruhdions  are  generated  from  a  rule  firing,  the  user 
needs  to  understand  the  concepts- a  rule  depends  on,  as  well  as  the  functions  that 
are  part  of  that  rule.  The  system  musrexplain  to  the  user  those  concepts  and  func¬ 
tions  the  user  does  not  already  know.  Furthermore,  if  the  user  does  not  understand 
the  more  fundamental  concepts  upon  which  the  understanding  of  a  given  concept 
is  dependent,  the  system  may  want  to  eiqilain  those  as  well.  For  our  example  con¬ 
cept,  recursion,  shown  in  Figures  5-4  and  5-7,  the  user  must  already  know  the 
concepts:  tests,  conditionals  and  functions  or  these  must  be  addressed  as  part  of 
the  strategy  for  explaining  recursion.  The  explanation  system  could  also  use  the 
domain  model  to  select  an  explanation  strategy  by  using  the  related-concepts  or 
related-functions  relationship  (slots).  In  the  case  of  recursion,  the  domain  model 
indicates  that  iteration  is  a  related  concept  so  one  explanation  strategy  would  be 
for  the  system  to  describe  recursion  as  compared  to  iteration. 
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Figure  5-6:  Concept  Layer  of  Domain  Model 
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(make-instance  ' lisp-concept 

rname  'recursion 
: dependent -on-concept  s 

' (tests  conditionals  functions) 

: related-concepts  ' (iteration) 

:level  'intermediate) 

Figure  5*7:  Clos  Specification  For  Concept  Recursion. 


The  user  modelling  component  uses  the  domain  model  for  two  purposes. 
The  model  provides  a  representational  basis  for  users’  knowledge;  the  user  model 
overlays  the  domain  model  to  capture  the  LISP  concepts  and  functions  that  a  user 
already  understands.  The  user  model,  as  will  be  discussed  fiirdier  in  Chapter  7,  is 
an  aimotadon  or  coloring  of  the  conceptual  graph  for  LiSP.  The  model  presently 
contains  an  in^licit  assumption  that  if  users  know  two  concepts  then  they  also 
know  about  any  relationships  them.  We  have  not  yet  considered  whether  this  is 
something  that  should  be  expUctty  representedand,  if  it  should  be,  whatmodifica- 
tions  to  the  domain  model  representation  might  be  required  to  accommodate  it. 

The  user  modelling  component  has  a  set  of  inference  methods,  again 
described  further  in  Clhapter  7,  that  build  up  individual  models  representing  each 
user.  Some  of  these  methods  use  the  structure  of  the  domain  model  as  a  basis.  For 
example,  when  the  system  determines  that  a  user  knows  about  recursion,  it  will 
annotate  the  user  model  with  an  assertion  that  the  user  understands  recursion,  and 
also  infer  that  the  prerequisites  are  known,  these  are  defined  in  the 
dependent-on-concepts  relation;  they  are  concepts:  tests,  conditionals,  and 
functions. 
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5.6.  Extending  the  Approach 

The  approach  to  modelling  LISP  described  here  is  not  a  unique  concep¬ 
tual  structuring  for  it  or  similar  domains.  Researchers  confronting  this  same 
problem  have  had  to  use  similar  formalisms  and  representation  languages.  At¬ 
tempts  to  develop  an  explanation  component  for  MYCIN  were  constrained  until  a 
representation  for  the  domain  other  than  the  inference  rules  was  used.  Wallis  and 
Shortlife  found  it  useful  to  describe  the  knowledge  representation  for  their  system 
in  terms  of  a  semantic  network  [Wallis,  Shortliffe  84].  Kass  used  the  NIKL 
representation  language  to  model  investment  knowledge  in  his  expert  adviser  so 
that  it  could  explain  advice  in  terms  appropriate  to  the  user’s  goals,  beliefs,  and 
prior  domain  knowledge  [Kass~88].  A  common  theme  in  this  research  is  that  there 
is  a  need  for  a  domain  representation  that  is  more  abstract  than  rules.  The  concep¬ 
tual  model  approach  presented  here  meets  die  requirements  in  the  domain  of  LISP 
for  a  critiquing  system;  it  could  possibly  be  used  for  a  larger  class  of  domains  and 
applications  as  well. 

The  ideal  knowledge  acquisition  qiproach  is  to  capture  deep  domain 
concepts  as  first  step  in  knowledge-based  system  development;  an  idea  diat  is  used 
in  the  explainable  expert  systems  (EES)  framework  [Neches,  Swartout,  Moore  85]. 
However,  it  is  far  easier  to  capture  procedural  knowledge  in  rule  form.  The  rule- 
based  paradigm  is  consistent  and  constrains  spedfrcation  of  knowledge;  this  as¬ 
sures  the  knowledge  engineer  that  the  system’s  actions  or  advice  will  {^ree  with 
true  human  expertise.  When  we  attenqrt  to  add  explanation  capabilities  the  rule- 
based  paradigm  breaks  down  and  second  order  domain  representations  become 
necessary. 


The  specific  concepts,  functions,  and  their  relationships  included  in  this 
implementation  may  not  be  universally  accepted.  We  found  that  experts  tie- 
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quently  do  not  agree  on  what  are  the  significant  concepts  undedying  LISP,  or  how 
they  relate  to  one  another.  The  model  implemented  here  was  developed  as  an  ap¬ 
proximate  consensus  of  what  makes  iq)  the  domain  of  LISP,  it  is  required  so  that 
we  can  determine  the  effectiveness  of  the  methods  that  generate  explanations  and 
model  users. 

5.7.  Summary 

This  chapter  described  a  domain  modelling  qjproach  and  implemen¬ 
tation  that  captures  LISP  knowledge  in  a  concq>tual  structure.  The  result  was  a 
graphical,  concept-based  domain  model.  The  motivation  for  having  the  model  is  a 
need  to  link  procedural  knowledge  already  contained  in  LlSP-CRmc  rules  with 
explanation  strategies  and  user  models  to  determine  how  to  accomplish  the  ex¬ 
planation  process.  The  types  of  entities  in  the  domain  model  are  USP  concepts, 
USP  functions  and  LiSP-CRmc  rules,  represented  as  nodes,  and  interconnected  via 
relationship  links.  Conceptual  gr£q}hs  provided  aa  s^ropriate  notation  for 
visualizing  and  capturing  the  domain  structure;  CLOS  was  used  as  the  implemen¬ 
tation  language.  The  qjproach  is  a  suitable  representation,  able  to  support  research 
on  e3q}lanation-giving  and  user  modelling.  Next,  we  will  describe  a  framework  for 
explanation  supported  by  this  domain  model. 


CHAPTER  VI 


THE  FRAMEWORK  FOR  EXPLANATION 

In  the  course  of  building  cooperative  knowledge-based  systems  one  ob¬ 
jective  is  take  advantage  of  the  difieient  strengths  of  users  and  counter  systems. 
The  system  provides  a  source  of  expert  domain  knowledge  that  is  used  to  make 
suggestions  to  users;  the  system  should  also  explain  those  suggestions.  Current 
explanation  systems  frequently  fail  to  satisfy  users  for  a  variety  of  reasons;  ex¬ 
planations  are  too  often  based  on  the  implicit  assumption  that  the  process  of  ex¬ 
plaining  is  a  one-shot  affair,  and  tharthe  system  will  be  able  to  produce  or  retrieve 
a  complete  and  satisfying  explanation  provided  ir  is  endowed  with  artificial 
intelligence.  Our  approach  takes^  advantage  of  informatiotrand  knowledge-based 
system  technology  already  available  to  provide  the  user  access  to  explanations  at 
different  levels  of  detail  and  complexity.  Developmental  efforts  in  this  work 
focused  on  the  concepts  to  be  e:q>lained,  rather  than  on  selecting  a  complete  pre- 
stored  explanation  appropriate  for  a  given  user.  The  domain  and  user  models 
provide  to  the  system  the  capability  to  determine  that  set  of  concepts. 

Early  research  on  how  to  explain  expert  knowledge  in  conputers  was 
done  in  the  context  of  MYCIN.  The  approach  taken  was  to  provide  a  rationale  by 
showing  users  an  historical  trace  of  the  rules  that  fired  in  arriving  at  a  diagnosis. 
Rule-tracing  explanatory  approaches,  even  when  “syntactically  sugared”,  to  make 
them  mote  readable,  are  difficult  to  follow.  Readers  of  that  literature  quickly  real¬ 
ize  that  anyone  not  familiar  with  medical  terminology  and  concepts  have  great  dif- 
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ficulty  understanding  them.  This  points  up  a  general  shortcoming  of  most  ex> 
planadon  approaches,  they  too  often  use  domain  concepts  there  readers  do  not 
know.  User  models  help  systems  to  overcome  this  shortcoming.  The  failure  of 
explanations  in  domains  more  closely  related  to  our  work  on  LlSP-CRmc  are  not 
difficult  to  locate.  A  standard  example  of  unsatisfactory  explanation  is  the  UNIX 
Man  Page  command. 

In  Figure  6-1  we  consider  a  more  realistic  example,  one  from  the  en¬ 
vironment  in  which  Lisp-Crttic  is  in^lemented,  and  from  a  system  generally  ac¬ 
knowledged  as  being  better  than  many  similar  documentation  systems.  If 
LiSP-CRmc  can  only  give  suggestions,  and  not  explain  those  suggestions,  dien 
users  might  attempt  to  achieve  an  understanding  of  the  transformation  by  using 
other  system  resources,  in  this  case  the  Symbolics  on-line  documentation.  The 
first  such  transformation  in  the  Chapter  4  scenario  recommended  replacing  a  cond 
with  an  if.  If  users  consult  the  Z>ocmnent  Examiner  for  information  about  these 
two  LISP  special  forms,  what  they  get  are  the  descr^ons  displayed  in  Figure  6-1. 
The  explanations  shown  are  better  than  most,  they  contain  examples;  begin  with 
one  or  two  sentence  minimal  explanation,  and  step-wise  expand  on  it;  they  use  a 
hypertext  display  that  allows  followup  and  further  exploration;  and  the  description 
for  i/even  refers  to  the  LISP  concqjt  which  it  exemplifies,  conditionals.  As  will  be 
shown,  the  explanations  still  fail  for  a  number  of  reasons.  In  general  they  are  too 
long,  attempting  to  cover  everything,  are  not  specific  to  the  user,  and,  in  this  case, 
have  to  be  viewed  individually  in  sequence  (they  are  only  shown  side-by-side  in 
this  figure  to  make  the  discussion  easier  to  follow),  this  makes  it  difficult  to  com¬ 
pare  the  two  and  come  up  with  the  rationale  for  why  they  might  be  interchangeable 
in  this  situation. 
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In  this  chapter  the  theoretical  understanding  of  explanation-giving  in 
cooperative  systems  will  be  discussed,  together  with  an  overview  of  related 
research.  It  will  also  cover  the  purposes  of  e:q>lanations  in  these  systems  and  the 
implications  for  the  user  model’s  role.  Finally,  an  explanation  framewodc  for 
LlSP-CRmc  will  be  described. 
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These  two  screen  displays  show  the  Document  Examiner  descriptions  for  if 
and  cond  that  are  retrieved  when  a  user  searches  for  information  on  the 
functions  in  the  cond-to-if  ndc.  Both  occupy  the  entire  viewer  and  on  a 
computer  screen  caimot  be  viewed  together  like  they  are  here. 

Figure  6-1:  Explanations  for  cond  and  if  bom  the  Document  Examiner 
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6.1.  Theory 

6.1.1.  The  Need  for  Explanations 

In  order  for  professionals,  managers,  and  scientists  to  accept  knowledge- 
based  systems,  it  is  essential  to  provide  explanations  of  the  knowledge.  The  need 
for  good  explanations  was  identified  in  a  study  of  physicians’  attitudes  towards 
expert  systems: 

Explanation.  The  system  should  be  able  to  justify  its  advice  in  terms  that  are  under¬ 
standable  and  persuasive.  In  addition,  it  is  prefer^le  that  a  system  adapt  its  explana¬ 
tion  to  the  needs  and  characteristics  of  the  user  (e.g.,  demonstrated  or  assumed  level 
of  background  knowledge  in  the  domain).  A  system  that  gives  dogmatic  advice  is 
likely  to  be  rejected.  [Teach,  Shortliffe  84,  p.  651] 

Explanation  in  cognitive  science  can  evoke  two  different  meanings:  the 
process  of  presenting  informatioir,  and  an  internal  cognitive  process  that  develops 
a  knowledge  representation.  Our  work  focused  on  the  presentation  process  while 
attempting  to  keep  both  meanings  in  mind.  The  internal-process  view  claims  that 
explanation  is  equivalent  to  understanding  [Schank  86].  According  to  this 
perspective,  humans  achieve  understanding  using  a  process  that  involves  generat¬ 
ing  their  own  explanations.  For  our  work  on  presenting  explanations  this  means 
that  systems  must  provide  the  information  that  is  needed  to  suj^rt  the  self¬ 
explanation  process.  If  systems  know  exactly  what  information  is  required  to  in¬ 
sure  understanding,  can  tailor  that  information  to  die  individual,  and  then  present  it 
in  an  optimal  form,  users  might  adopt  it  as  their  own.  Such  a  goal  for  computer¬ 
generated  explanations  (or  even  those  produced  by  another  human,  for  that  matter) 
is  too  ambitious.  Rather  than  attempting  complete,  ideal  explanations  which  each 
user  can  understand  and  integrate  into  their  mental  models,  computers  must  in¬ 
stead  concentrate  on  providing  users  with  the  material  required  to  produce  their 
own  explanations.  This  means  generating  explanations  with  the  computer  not 
merely  displaying  stored  ones;  explanations  that  use  the  domain  concepts 


propriate  to  a  particular  problem  solving  context,  so  as  to  provide  users  an  oppor¬ 
tunity  to  produce  a  self-explanation,  and  therefore  achieve  understanding. 

6.1.2.  Functions  for  Explanations 

We  are  investigating  how  to  design  systems  that  serve  users  actively 
engaged  in  their  own  work  —  cooperative  problem  solving  systems  that  provide  a 
task-based  environment  in  which  users  woik  toward  goal  accomplishment.  Sys¬ 
tems  that  support  users’  work  are  more  than  media  used  for  describing  their 
problems,  and  more  than  just  tools  to  extract  useful  information  from  a  database. 
They  should  be  active  agents  that  provide  for  problem-domain  communications  at 
the  construction  artifact  level  [Fischer,  Lemke  88],  can  critique  users  work,  and 
are  able  to  explain  their  knowledge.  We  analyzed  the  reasons  users  seek  explana¬ 
tions  and  determined  that  a  common  triggering  condition  is  experiencing  some  sort 
of  impasse.  A  similar  idea  motivates  theory  about  what  should  h£^>pen  during  in¬ 
structional  episodes,  there  the  emphasis  is.  on  determining  how  to  communicate 
knowledge  to  overcome  impasses,  and  how  students  formulate  new  procedural 
knowledge  [VanLehn  88].  We  cataloged  these  as  ’’task-oriented  impasses”  in  or¬ 
der  to  develop  a  better  understanding  of  where  explanation  fits  in  each  simadon. 
There  are  four  categories  of  task-oriented  impasses: 

1.  Action  impasses  occur  when  users  do  not  know  what  to  do  next. 

Some  action  impasse  questions  are:  What  should  1  do  next?  Is  action 
the  right  thing  to  do  next?  How  do  I  do  action!  What  did  the  system 
just  do?  What  are  the  results  of  doing  action!  Can  I  do  action  now? 

These  are  the  types  of  impasses  help  systems  should  be  designed  to 
address. 

2.  A  communication  impasse  is  a  failure  to  understand  a  given  object  in 
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the  environment.  Representative  questions  are:  What  is  object^ 

Why  is  objectl  shown  instead  of  object2?  What  is  the  rationale  for 
suggesting  object  or  action!  This  is  the  category  of  impasse  the  user 
experiences  in  the  scenario  when  trying  to  decide  whether  to  accept 
or  reject  a  suggestion. 

3.  Motivation  impasses  fall  into  the  realm  of  behavioral  psychology; 
their  basis  is  an  anthropomorphic  view  of  the  computer  system. 
Representative  questions  are:  Why  did  the  system  do  action!  Why 
did  the  system  just  communicate  with  me?  Why  did  the  system  just 
sayX?  Why  should  I  do  action? 

4.  Curiosity  impasses  are  a  bit  different.  The  other  categories  consist  of 
questions  that  arise  when  users  encounter  a  problem.  Curiosity  im¬ 
passes  are  not  necessarily  impasses,  in  a  strict  sense,  but  rather  are 
diversions.  They  are  circumstances  in  which  users  gather  infor¬ 
mation  that  is  interesting  or  helpful,  but  if  it  is  missing,  furdier  work 
is  not  actually  impaired.  For  consistency,  these  are  also  “im¬ 
passes".  Questions  that  illustrate  curiosity  impasses  are:  Is  object  a 
concept  X?  How  do  object j  and  object2  differ?  On  occasion, 
LlSP-CRmc  users  also  experience  these;  it  is  a  case  where  they  un¬ 
derstand  the  suggestion  but  see  it  as  an  opportunity  to  improve  upon 
their  knowledge  and  therefore  request  explanations  so  as  to  engage 
in  active  exploration. 

Assisting  users  during  problem  solving  requires  that  explanations  be 
designed  to  help  them  overcome  impasses.  Such  explanations  in  cooperative 
problem  solving  systems  can  serve  four  functions.  We  adapted  these  functions  for 
cooperative  problem  solving  from  ones  that  were  found  during  investigations  sur- 
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rounding  MYCIN  in  which  they  studied  users  of  knowledge-based  medical  infor¬ 
mation  systems  [Wallis,  Shortliffe  84]. 

1.  Explanations  allow  users  to  examine  the  system’s  recommendations. 

2.  Users  need  explanations  to  relate  recommendations  to  domain  con¬ 
cepts  —  to  understand  “what  is  suggested”. 

3.  The  explanation  should  help  users  to  see  the  rationale  for  recommen¬ 
dations  —  to  understand  "why  this  would  be  better". 

4.  Explanations  are  needed  by  users  to  learn  the  underlying  domain 
concepts. 

These  functions  are  not  mutually  exclusive;  single  explanations  in  a  cooperative 
problem  solving  system  will  have  to  accommodate  multiple  purposes. 

6.1.3.  Shortcomings  of  Current  Approaches 

Most  attempts  to  provide  explanations  use  prestored  scripts  in  the  form 
of  carmed  text.  Those  types,  of  descriptions  have  been  criticized  as  difficult  to 
understand,  incomplete,  and  hard  to  navigate  [Weiss  88].  Empirical  studies  of 
tutoring  in  both  humans  and  con^uters  determined  that  canned  explanations  are 
insufficient  approaches  [Fox  88].  Because  critiquing  and  tutoring  are  closely  re¬ 
lated,  many  of  the  problems  listed,  there  apply  to  critiquing  as  well.  “Canned 
text”  is  intended  to  meant  pre-written  text,  stored  in  machine  memory  in  a  form 
that  the  system  cannot  interpret  meaningfully  (most  likely  as  character  strings.) 
The  use  of  canned  explanations  is  not  inherendy  bad  just  because  it  is  done  by  a 
computer.  Empirical  studies  of  human  explanations,  found  a.  similar  strategy  is 
often  employed  by  people  when,  explaining  something  “for  the  sake  of  others” 
[Schank  86].  The  difference  is  that  people  understand  their  prestored  explana¬ 
tions  —  they  make  sense  to  the  explainer;  they  represent  a  form  of  mental  model. 
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When  it  happens  that  the  recipient  does  not  understanding  such  an  explanation,  the 
nature  of  most  human-to-human  discourse  allows  them  to  query  the  explainer  for 
clarification  or  elaboration. 

Canned  explanations  captured  in  computer  systems  are  inadequate  when 
their  content  is  poorly  chosen  or  presented.  There  are  five  primary  reasons  for  the 
failure  of  computer  explanation  q>proaches: 

1.  Explanations  are  too  long;  users  get  lost,  bored,  or  confused;  they  do 
not  bother  reading  the  text  just  to  find  what  they  need.  This  is  espe¬ 
cially  characteristic  of  many  on-line  help  systems. 

2.  Too  often,  explanations  attempt  to  tell  the  user  everything  they  could 
possibly  need  to  know  rather  than  determining  what  is  specifically 
required  for  the  situation  at  hand,  and  for  the  individual  requesting 
the  explanation.  This  creates  complexity  and  is  firequently  what 
makes  them  too  long. 

3.  Users  are  not  provided  the  c^ability  to  ask  follow-up  questions  or 
enter  into  a  dialog  with  the  computer.  The  explanations  are  designed 
as  if  they  could  satisfy  their  reader  with  a  single  presentation. 

4.  The  explanations  do  not  provide  examples  to  facilitate  understanding 
textual  descriptions.  Even  when  available,  examples  may  be  in¬ 
appropriate  for  the  user’s  particular  problem. 

3.  The  explanation  text  is  written  from  an  author’s  perspective.  It  is 
based  on  that  author’s  conceptual  model  of  the  domain,  not  the 
readers’. 

The  exanples  from  the  Document  Examiner  shown  in  Figure  6-1  exhibit  some  of 
these  characteristics.  They  fail  on  the  first  account,  being  longer  than  most  users 
would  want  in  the  cond-to-if  transformation-situation.  The  system  does  not,  of 
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course,  individualization  the  descriptions,  therefore  they  also  fail  on  the  second 
account  Document  Examiner  does  provide  hypertext  capabilities,  therefore  a 
limited  form  of  follow-up  is  provided.  Also,  both  doounentation  entries  (as  do 
quite  a  few  in  Document  Examiner)  contain  multiple  examples  to  facilitate  under¬ 
standing.  On  the  last  point,  the  explanation  of  cond  is  particularly  poor,  it  appears 
to  have  been  written  by  a  LISP  expert  (hacker)  from  his  or  her  individual  perspec¬ 
tive;  the  one  for  if  is  actually  much  better;  its  author  attempted  to  direct  it  toward  a 
less  sophisticated  programmer. 

Some  systems  attempt  to  overcome  several  of  these  problems,  but  none 
addresses  all  shortcomings.  Our  strategy  is  to  recognize  the  shortcomings  while 
using  an  interactive  approach  based  on  available,  technology  integrated  with 
domain  and  user  modelling  capabilities.  We  consider  the  limitations  of  canned 
text  but  try  to  be  realistic  about  current  capabilities  of  computer  systems.  In 
developing  explanation  strategies  there  is  too  often  an  assumed  environment  which 
contains  an  intelligent  computer  able  to  generate  natural  language,  predict  users’ 
needs,  and  enter  into  a  followup  dialog.  Techniques  are  needed  now  that  woik 
within  the  constraints  of  available  technology.  Based  on  these  limitations,  an  ex¬ 
planation  framework  was  developed;  it  considers  what  is  possible;  a  part  of  it  was 
implemented  to  further  our  understanding  and  evaluate  the  role  of  the  user  model. 

6.1.4.  Basis  for  Minimalist  Explanations 

If  a  user  model,  such  as  the  one  described  in  Chapter  6,  can  provide  a 
detailed  representation  of  users’  knowledge,  then  it  will  be  possible  to  formulate 
and  present  an  appropriate  explanation.  One  method  to  achieve  that  is  the  min¬ 
imalist  approach  [Fischer  et  al.  90].  The  ideas  for  minimal  explanations  share  the 
underlying  theoretical  foundations  with  minimal  approaches  to  instmction 
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[Carroll,  Carrithers  84].  Both  use  the  principle  that  an  optimal  first  approach  is  to 
provide  users  with  the  minimum  amount  of  information  required  to  accomplish 
their  task.  Theoretical  bases  for  this  ^jproach  are  found  in  related  woik  on  dis¬ 
course  comprehension: 

1.  Short-term  memory  is  a  fundamental  limiting  factor  in  reading  and 
understanding  text  [Dijk,  Kintsch  83;  Britton,  Black  85].  The  best 
explanations  are  those  that  contain  no  more  information  than  ab¬ 
solutely  necessary,  since  extra  words  increase  the  chances  that  essen¬ 
tial  facts  will  be  lost  from  memory  before  the  entire  explanation  is 
processed. 

2.  It  is  important  to  relate  written  text  to  the  readers’  existing 
knowledge  [Kintsch  89;  Fischer  et  al.  88]. 

Similar  practical  guidelines  are  also  found  in  the  theory  of  rhetoric. 
Flesch  developed  formulae  to  evaluate  die  readabilit3rof  text  [Flesch  49]  which  arc 
fcequently  used  to  evaluate  documentation  and  instraction.  Computer  explanation 
systems  should  comply  with  similar  standards;  using  short  sentences  and  known 
vocabulary  are  important  criteria.  Stnuik  and  White’s  guide  to  good  writing  con¬ 
tains  similar  advice;  they  tell  writers  “Don’t  explain  too  much’’  when  writing  ex¬ 
planatory  text  [Strunk,  White  57]. 

6.2.  Related  Work 

Some  research  on  explanations  in  knowledge-based  systems  assumes  a 
natural  language  interaction,  such  as  in  the  dialog  advisory  systems  discussed  in 
Chapter  2.  Another  approach  also  attempts  to  capture  expertise  during  the 
knowledge  acquisition  phase  of  building  an  expert  system;  that  approach  uses  a 
methodology  which  will  later  facilitate  explaining  that  knowledge:  the  explainable 
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expert  system  q^proach  (EES)  developed  by  Swartout  [Swartout  83]  is  one  ex¬ 


ample. 

The  fimdamental  claim  behind  EES  is  that  explanation  is  simplified  if 
the  knowledge  acquisition  process  occurs  at  the  conceptual  level  and  a  system 
automatically  generates  the  operational  knowledge  (Le.,  rules).  Then  to  explain  a 
rule,  the  system  can  trace  through  diat  portion  of  the  conceptual  domain 
knowledge  from  which  the  rule  was  generated.  That  approach  is  appealing  but  has 
not  been  enthusiastically  accepted  as  standard  knowledge-engineering  practice. 
LlSP-CRmc’s  rule  base  was  developed  using  the  traditional  knowledge  acquisition 
process  of  querying  expert  LISP  programmers.  For  systems  to  explain  something 
captured  in  procedural  (rule-based)  form  requires  reverse  engineering  of  the  ap¬ 
plicative  knowledge  (in  our  case  die  transformation  rules)  in  order  to  determine  the 
concepts  behind  each  rule.  The  process  followed  in  developing  and  refining 
LiSP-CRmc,  as  opposed  to  the  one  proposed  by  EES,  is  mote  indicative  of  what 
will  be  the  standard  approach  for  providing  knowledge-based  systems  with  ex¬ 
planation  capabilities,  for  the  near  future. 

Moore  extended  the  EES  work,  in  a  program-transformation  system 
similar  to  LiSP-CRinC  [Neches,  Swartout,  Moore  85].  Her  specific  research  ad¬ 
dressed  a  situation  where  users  need  to  follow-up  on  explanations  with  clarifica¬ 
tion  questions.  Her  ‘‘reactive”  approach  provides  the  user  with  an  initial  explana¬ 
tion,  but  accommodates  the  situation  where  it  fails  to  satisfy  the  user,  it  provides 
increasingly  informative  fail-back  explanations  [Moore  87].  Her  framework  ach¬ 
ieves  a  fall-back  capability  by  monitoring  and  recording  the  dialog  between  the 
system  and  users,  then  using  this  dialog  trace  to  identify  and  overcome  difficulties. 
Moore  still  agrees  with  our  goal  [Moore  89],  that  a  convivial  system  should  make 
a  good-faith  effort  to  provide  the  right  explanation  the  first  time;  it  is  when  that 
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fails  that  her  reactive  approach  or  something  similar  is  needed.  Providing  the  best 
possible  initial  explanation  requires  the  system  to  understand  the  domain  at  a  level 
that  supports  the  generation  process  and  to  be  capable  of  modelling  its  users.  Her 
approach  holds  promise  for  future  generations  of  knowledge-based  systems  but 
depends  too  heavily  on  natural  language  generation  and  dialog  management.  Until 
such  equabilities  are  commonplace,  other  available  techniques  should  be  exploited. 
Whether  a  powerful  access  technique,  such  as  hypermedia,  or  a  dialog  manage¬ 
ment  approach  to  supporting  fallback  requirements  is  the  better  approach  will  only 
be  determined  when  both  have  matured  to  the  point  where  they  can  be  subjected  to 
conq>arative  evaluations. 

Several  efforts  to  provide  computer-based  explanations  generate  strings 
of  natural  language.  Some  rely  on  a  user  model  for  tailored  explanations  [Kass 
87b]  while  others  generate  the  same  explanation  for  any  user  [Danlos  87;  Water¬ 
man  etal.  86].  The  natural  language  approach  is  conq>lex  and  difficult,  and  the 
syntactic  formats  are  limited. 

Natural  language  approaches,  such  as  Kass’s  reliance  on  Grice’s  rules 
for  cooperative  conversation  and  Moore’s  iterative  fail-backs,  use  human-to- 
human  discourse  as  their  model  for  human-computer  communication.  This  may 
be  unreasonable,  especially  given  the  difficulties  of  reading  large  sections  of  text 
from  a  CRT  screen  [Hansen,  Hass  88].  Knowledge-based  system  designers  need 
to  recognize  the  special  capabilities  and  limitations  of  computers  rather  than  trying 
to  coerce  the  natural  language  paradigm  into  a  screen-  and  keyboard-interaction 
style  [Kennedy  etal.  88;  Fischer  88b].  Another  crucial  issue  in  explanation, 
whether  between  humans,  or  between  a  hunuin  and  a  computer,  is  not  natural  lan¬ 
guage,  but  using  all  of  the  available  interaction  facilities  to  insure  that  users  are 
comfortable  with  the  concepts  presented  during  the  explanation  process;  the  es¬ 
sence  of  natural  communications. 
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Paris  [Paris  87;  Paris  89]  developed  an  q^proach  to  explanation  based  on 
an  assumed  user  model.  Her  work  provided  initial  motivarion  for  our  user  modell¬ 
ing  investigations  [Mastaglio  90b].  She  developed  a  theory  that  builds  hybrid  tex¬ 
tual  descriptions  for  devices  using  two  strategies,  a  process  trace  and  a  con¬ 
stituency  scheme. 

•  A  process  trace  describes  how  an  object  works  (her  research  was  in¬ 
terested  in  explaining  mechanical  and  electronic  artifacts  like  the 
telephone.) 

•  A  constituency  scheme  describes  an  object  in  terms  of  its  con^nent 
parts  (like  the  receiver,  transmitter,  etc.  of  the  telephone.) 

A  hybrid  explanation  for  a.  device  is  actually  a  mixture  of  the  two  methods  based 
on  what  users  already  know.  Those  constituents  with  which  a  user  is  familiar  need 
only  be  indicated  as  component  parts  of  the  device  being  described,  but  others 
need  to  be  explained  in  terms  of  how  they  operate  (their  process),  or  their  own 
constituents,  and  so  on.  The  process  recursively  executes,  capturing  those  portions 
of  the  domain  (objects  or  concepts)  that  an  individual  user  needs  explained  to  un¬ 
derstand  the  device.  A  user  model  will  indicate  to  her  system  which  concepts  and 
specific  items  in  the  knowledge  base  the  user  already  knows.  That  information 
will,  in  turn,  guide  explanation-generation,  combining  the  two  strategies  to  insure 
that  the  explanations  are  presented  at  a  level,  and  in  terms  of  concepts,  that  users 
already  understand.  Her  scheme  can  be  used  to  generate  an  explanation  for  a 
lJSP-(3RrnC  rule  in  terms  of  the  LISP  concepts  and  functions  underlying  the  rules 
in  the  knowledge-base.  LISP  concepts  are  equivalent  to  the  underlying  concepts 
her  model  uses,  and  LISP  functions  are  analogous  to  specie  items  in  the 
knowledge  base.  This  approach  is  a  candidate  strategy  for  use  in  the  rinal  steps  of 
presenting  an  explanation  to  a  user. 
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Requiring  a  knowledge-based  system  to  have  a  concept  level  domain 
motfel  is  not  a  unique  finding.  Oiandrasekaran  and  associates  investigated  the 
need  for  deep  domain  models  in  expeit  systems  [Chandrasekaran,  Tanner,  Joseph- 
son  88;  Chandrasekaran,  Tanner,  Josephson  89].  Their  theoretical  framework 
claims  that  explanation  involves  three  issues:  presenting  the  explanation,  modell¬ 
ing  the  user,  and  endowing  a  system  widi  “self-understanding”.  Their  research 
focuses  on  the  third  issue.  Their  solution  is  similar  to  Swartout’s  in  that  they 
propose  a  “generic  task  methodology”  approach  to  building  expert  systems.  The 
paradigm  focuses  at  the  level  of  the  task  rather  than  that  of  abstract  domain  con¬ 
cepts;  it  makes  basic  explanation  constructs  available  at  a  level  of  abstraction 
closer  to  the  user’s  conceptual  level,  it  is  similar  to  the  work  on  human  problem- 
domain  communications  [Fischer,  Lemke  88].  It  also  appeals  to  general  domain 
knowledge  in  order  to  justify  the  system’s  problem-solving  ^jproaches. 

In  a  perspective  of  what  is  h^jpening  to  die  user  cognitively,  one  could 
consider  explanations  to  be  foims  of  knowledge  retained  in  long  term  memory, 
and  later  reused  to  provide  situational  understanding.  This  is  the  basis  for 
Anderson’s  work  on  learning  by  analogy  [Anderson,  Thompson  86],  and  has  been 
investigated  by  Lewis  as  a  substitution  process  [Lewis  89].  This  theoretical  view 
could  be  used  by  a  system  to  chose  a  strategy  for  presenting  explanations  based  on 
a  user  model  containing  a  record  of  either  the  exact  explanations  previously 
received,  or  chunk-size  domain  entities  (e.g.,  critic  rules  or  lisp  concepts)  that  were 
the  focus  of  explanations;  either  of  these  could  be  used  as  a  starting  point  for  an 
analogical  explanation.  In  the  scenario,  the  user  generates  his  own  explanation  for 
a  rule  {cond-erase-tjiil)  using  this  process;  this  rule  is  similar  to  one  previously 
explained  (cond-erase-pred.t),  and  the  user  can  forego  requesting  one  from  the 
system.  The  user  model  also  needs  to  represent  the  user’s  possible  goals:  goals 
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related  to  improving  the  immediate  piece  of  code  (e.g.,  make  it  easier  for  other 
programmers  to  read),  or  goals  related  to  learning  LISP  (e.g.,  they  want  to  become 
proficient  users  of  the  language).  This  woik  has  not  explored  methods  for  infer¬ 
ring  user  goals  or  plans,  but  provides  for  goal  representation  in  the  user  model. 
Goal  and  plan  recognition  is  a  significant  research  problem  in  itself. 

En^irical  observations  of  problem-solving  interactions  between  sales¬ 
men  and  customers  in  a  large  hardware  store,  observed  that  explanation  never  took 
the  man  page  2^roach.  When  explanations  were  required,  the  approach  was  one 
of  minimizing  the  explanation,  then  following  up  on  imclear  concepts  when  neces¬ 
sary  [Reeves  90].  This  is  interesting  if  you  consider  the  fact  that  the  particular 
store  carries  over  350,000  different  items  in  over  33,000  square  feet  of  retail  space. 
If  salesmen  took  the  s^roach  found  in  many  computer  systems,  the  explanations 
given  would  be  extraordinarily  long,  to  insure  completeness,  and  complex,  in  order 
to  account  for  relationships  to  other  items  in  the  store. 

Argumentation  is  another  approach  to  facilitating  user  understanding  of 
the  domain  knowledge  behind  a  critique  or  suggestiom  Impressive  results  have 
been  achieved  using  the  argumentation  ^roach  in  a  critic  for  kitchen  design 
[Fischer,  McCall,  Morch  89b].  Argumentation,  as  used  in  paradigms  such  as 
issue-based  information  systems  (IBIS),  provides  a  context  for  exposing  the  issues 
underlying  a  given  suggestion.  Argumentation  approaches  do  not  try  to  provide 
information  at  an  appropriate  level.  Users  of  these  systems  retain  primary  respon¬ 
sibility  for  traversing  the  issue  base.  It  is  possible  that  they  will  find  it  difficult  to 
locate  exactly  what  they  need,  or  to  understand  it,  when  the  complexity  level  is  not 
adjusted  to  their  individual  expertise. 


118 

63.  An  Explanation  Framework  to  Support  Critiquing 

To  support  explanation  in  critics  requires  sufficient  knowledge  on  the 
part  of  the  system  to  describe  what  is  going  on  and  why.  Operational  knowledge 
in  LiSP-CRrnC  is  captured  in  transformation  rules.  For  users  to  understand  a  trans¬ 
formation,  they  need  to  know  the  LISP  fimctions  in  the  rule  and  the  concepts  that 
makes  it  valid.  This  was  informally  observed  during  usability  testing  on  the 
second  version  of  LiSP-CRmc  when  we  attempted  to  satisfy  users  with  canned- 
text  generic  explanations  of  die  rationale  for  each  rule. 

The  domain  model  provides  a  conceptual  basis  for  an  explanation  in 
terms  of  those  functions  and  concepts  that  are  prerequisite  knowledge.  Determin¬ 
ing  prerequisite  knowledge  is  a  recursive  process  because  understanding  those 
domain  concepts  that  are  prerequisites  for  the  given  concept  requires,  in  turn,  un¬ 
derstanding  their  prerequisites  and  so  on.  To  support  such  an  approach,  the  deep 
structure  in  the  domain  model  is  queried  to  obtain  a  comrepr-ser  comprised  of 
those  prerequisites.  A  satisfactory  eiqilanation  tqiproach  needs  to  still  do  some¬ 
thing  more,  it  must  identify  the  concepts  in  that  set  that  do  not  require  explaining^ 
because  the  user  already  knows  them.  Furthermore  it  will  reason  about  the  best 
way  to  explain  the  remaining  concepts. 

We  investigated  ways  to  organize  explanations  for  a  system  such  as 
USP-CRITIC  and  developed  a  firameworic  that  includes  different  levels  for  explana¬ 
tions  (shown  in  Figure  6-2).  The  explanation  levels  capture  necessary  and  suf¬ 
ficient  conditions  for  adequate  explanations.  Each  level  incrementally  enhances 
work  done  at  a  lower  level,  integrating  additional  knowledge  about  the  user  and 
the  domain.  A  Level  0  explanation  does  not  require  knowledge  about  individual 
users.  It  uses  the  domain  model  to  meet  a  necessary  condition  —  knowing  what  to 
explain.  The  explanation  component  is  provided  the  set  of  prerequisite  concepts 
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required  to  understand  an  object  needing  to  be  explained.  Level  1  brings  the  user 
model  into  the  process;  here  the  prerequisite  set  of  concepts  is  “filtered”  through 
the  tiser  model  to  determine  the  subset  qjpropriate  for  a  given  individual.  In  many 
cases,  that  filtered  set  is  probsU^ly  larger  than  we  want  to  explain  in  a  single 
episode.  Therefore,  at  Level  2  the  explanation  component  needs  to  know 
strategies  that  detennine  exacdy  which  of  that  subset  to  explain  and  how. 


Five  levels  of  explanation  are  identified.  Level  0  insures  all  prerequisite 
knowledge  for  a  given  domain  object  is  available  to  die  explanation  com¬ 
ponent.  Level  1  builds  on  level  0  and  so  forth.  The  current  LISP-CrtitC 
system  provides  simplified  level  2  explanations.  Level  3  and  4  will  require 
presentation  and  nati^  language  generation  techniques. 

Figure  6-2:  Explanation  Levels 


A  system  operating  at  Level  2  passes  a  sufficiency  test:  it  knows  what 
concepts  to  explain  to  an  individual  user  in  a  specific  situation.  However,  it  is  still 
faced  with  the  presentation  problem;  explanations  need  to  be  presented  in  a  man¬ 
ner  and  style  that  will  make  them  more  readable.  Achieving  this  level  means  the 
system  will  need  to  make  use  of  additional  domain  knowledge  or  other  infor¬ 
mation  in  the  user  model,  in  order  to  determine  a  “best”  strategy  for  explaining  a 
concept  For  example,  a  system  could  make  use  of  the  related  links  in  the  domain 
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model  and  the  user  model  contents  to  determine  candidate  concepts  or  functions 
for  use  in  a  differential  description  [Fischer  et  al.  90];  one  object  can  be  described 
differentially  in  terms  of  another  that  the  user  already  knows.  Another  example 
would  be  to  apply  user  goals  captured  in  the  user  model  to  determine  the  strategies 
that  support  those  goals.  Level  4  performs  “syntactic  sugaring.”  Here  the  in¬ 
dividual  explanations  from  Level  3  are  ordered  and  appropriately  linked,  a  non¬ 
trivial  process  that  requires  the  system  to  have  knowledge  of  discourse  as  well  as 
natural  language  generation  ctqjabilities. 

6.4.  Role  of  the  User  Model  in  Explanations 

The  user  model  is  discussed  in  the  next  chapter  but,  because  its  stated 
purpose  is  to  support  explanation  generation  it  is  important  to  consider,  in  the  con¬ 
text  here,  what  criteria  for  the  user  noodel  are  established  by  this  explanation 
framework.  This  section  will  sununarize,  and  review,  the  insights  for  the  user 
modelling  cotrqHjnent  that  resulted  from  the  analysis  of  the  explanation  process. 

Cooperative  systems  must  tailor  their  explanations  to  individuals  in  or¬ 
der  to  accommodate  adequately  the  four  functions  previously  listed.  The  system 
needs  a  basis  for  tailoring:  this  is  the  role  of  the  user  model.  One  simplified  ap¬ 
proach  is  to  classify  users  by  their  expertise  (e.g.,  novice,  intermediate,  expert); 
but,  as  reported  earlier,  this  is  not  a  valid  representation  for  many  domains  and 
users.  A  finer  grained  representation  that  follows  from  Paris’s  wo±,  represents 
user’s  knowledge  in  terms  of  the  domain  objects  and  concepts. 

The  user  model  needs  a  representation  of  the  user’s  domain  knowledge 
detailed  enough  to  support  each  of  the  five  "levels”  of  explanation  shown  m 
Figure  6-2.  It  has  to  be  based,  at  least  in  part,  on  the  conceptual  model  of  the 
domain,  so  that  it  can  filter  the  set  of  concepts  that  form  the  explanation  basis. 
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The  model  needs  to  capture  the  user’s  goals  in  order  to  support  Level  3  explana- 
tiokis.  Programmers  who  use  USP-CRmc  have  goals  of  either  making  their  code 
easier  for  others  to  read  (such  as  in  the  scenario)  or  making  it  execute  more  ef¬ 
ficiently.  A  subsuming  goal  for  both  is  learning  to  produce  better  code,  the  type  of 
optimization  goal  merely  determines  the  dimension  along  which  they  want  to  learn 
to  improve  their  programs.  Higher  level  goals,  such  as  learning  how  to  use  new 
programming  stmctures  that  can  make  programs  better  from  the  start,  need  to  be 
acquired  through  explicit  questioning.  Problem-specific  goals  (such  as  writing  a 
function  to  factorial))  are  not  within  the  scope  of  the  current  system.  LlSP-CRmc 
does  not  know  how  to  achieve  these  problem  specific  objectives;  it  is  neither 
capable  of  automatic  programming,  nor  does  it  have  the  knowledge,  envisioned  for 
the  Design  Apprentice  portion  of  the  the  Programmer’s  Apprentice.  Design  Ap¬ 
prentice  knows  how  to  automatically  select  the  low-level  program  cliches  ap¬ 
propriate  for  achieving  a  specified  objective  [Rich,  Waters  90]. 

Supporting  Level  4  explanations  is  more  difficult  because  they  involve 
solving  difficult  issues  on  the  research  agenda  for  both  explanation-giving  and 
natural-language  generation.  We  will  not  know  all  requirements  for  user  models 
to  support  this  level  until  that  leseardi  matures.  It  is  possible  to  conjecture  some 
important  capabilities,  such  as,  knowing  the  education  and  reading  comprehension 
level  of  users,  because  that  knowledge  could  guide  the  generation  of  an  ap¬ 
propriate  explanation.  To  make  the  scenario  in  Ch^ter  4  more  realistic,  this  type 
of  higher  level  processing  was  manually  applied  to  create  the  mock-up  explana¬ 
tions.  The  system  cannot  presently  generate  anytiiing  that  complex,  it  can  only 
display  short  explanations  for  the  selected  concepts. 

In  its  present  form,  the  user  model  that  was  developed  provides  partial 
support  for  explanation-giving  according  to  the  presented  framework.  One  sup- 
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ported  approach  is  the  minimal  explanation  strategy;  it  interrogates  the  user  model 
to  determine  a  minimal  set  of  concepts  to  explain.  Such  strategies  are  possible 
because  the  model  knows  which  concepts  are  familiar  to  the  user. 

6S.  LiSP-CRmc  Explanation  System 

A  conceptual  overview  of  the  explanation  con^nent  shows  how  it  con¬ 
forms  to  the  framewoik  discussed  above.  One  focus  in  this  implementation  is  to 
use  information  already  available  to  the  system,  and  to  present  that  information 
using  ideas  which  provide  the  best  support  for  users'  needs. 

There  are  several  sources  of  information  that  is  already  available  to  the 
system  and  which  can  be  used  to  satisfy  some  explanation  requirements.  This  in¬ 
formation  is  presented  using  techniques  that  were  designed  to  provide  users  access 
to  the  information  in  four  layers  of  increasing  detail.  These  layers  help  to  visual¬ 
ize  how  the  system  is  designed  and  operates;  they  should  not  be  confused  with  the 
conceptual  levels  shown  in  Figure  6-2. 

1 .  A  fundamental  piece  of  information  is  the  name  of  the  rule  that  is  the 
basis  for  a  transformation.  The  rule  name  is  an  abstract  reference  to 
a  chunk  of  domain  knowledge,  in  the  domain  model  that  chunk  is  an 
instance  of  the  class  Icr-rule,  and  it  may  or  may  not  have  meaning  to 
users.  When  it  does  have  meaning,  users  may  be  satisfied  just  by 
knowing  which  rule  fired  and  further  explanations  may  not  be  re¬ 
quired.  An  example  of  this  occurred  in  the  third  scenario  episode 
when  the  user  recognized  the  cond-to-if  rule  because  it  had 
previously  been  explained.  We  are  assuming  that  the  name  evoked 
the  appropriate  conceptual  understanding  on  his  part,  seeing  the  two 
versions  of  the  code  may  have  also  played  a  role. 
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2.  That  second  piece  of  information  is  precisely  those  two  versitHis  of 
the  code.  The  user  can  compare  the  system-transformed  code  with 
his  own.  The  system  displays  the  user’s  code  together  with  the  sug¬ 
gested  changes.  Sometimes  this  also  triggers  an  understanding  of  the 
underlying  concepts  and  rationale  for  the  transformation.  This  is 
what  occurred  when  die  cond-erase-t.ml  rule  fired  in  the  second 
scenario  and  no  explanation  was  required.  A  rule  based  on  similar 
concepts  had  just  been  explained  and  the  user  can  generate  his  own 
understanding. 

3.  The  minimal  explanation  layer  is  the  point  where  empirical  obser¬ 
vations  come  into  play.  The  user  is  provided  with  a  text  description 
of  the  system’s  advice  based  on  the  xmderlying  concepts  in  the 
domain.  The  text  descrqjtion  is.  comprised  of  portions,  of  hypertext 
associated  with  each  domain  concept  and  rule. 

4.  A  hypertext-based  information  space  is  also  part  of  the  underlying 
computational  environment  LiSP-CRmc  provides  access  to  this  in¬ 
formation  as  a  source  of  additional  information  for  users  who  want 
to  know  mote  or  who  are  not  satisfied  with  tho  minimalist  explana¬ 
tion  of  the  advice.  Users  navigate  through  the  hypertext  space  only 
after  the  system  locates  them  within  it  in  an  appropriate  context. 

The  explanations  in  Layers  3  and  4  require  information  from  the  user 
model  to  tailor  their  presentations  to  each  individual  user.  The  user  modelling 
component  can  tell  the  explanation  component  which  concepts  users  already  im- 
derstands  so  the  system  can  avoid  telling  them  what  they  already  know. 

Layer  4  explanations  back  up  the  minimal  explanations  with  access  to 
more  detailed  information.  The  system  uses  hypermedia  along  widi  some  other 
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of  these  techniques  are  inter-referential  input/output  [Draper  86];  command 
completion  (the  user  can  type  abbreviations  and  the  system  completes  the  com¬ 
mand);  and  mixed  initiative  dialogue  (either  participant  can  take  the  initiative  or 
volunteer  information)  [Carbonell  70]. 

The  approach  used  in  the  current  implementation  evolved  from 
rule-tracing  and  carmed  text  explanations  methods  attempted  in  an  earlier  version 
of  LlSP-CRmC  [Frank,  Lyim,  Mastaglio  87].  Alternative  carmed  explanations  for 
each  rule  were  provided;  each  designed  to  accommodate  a  particular  level  of  ex¬ 
pertise.  To  chose  the  correct  explanation,  the  system  had  to  classify  a  user  as  a 
novice,  intermediate  or  expert  programmer.  No  user  model  acquisition  was  ac¬ 
tually  attempted  because  system  testing  using  protocol  studies  and  observations  of 
users  pointed  out  that  the  explanation  £q>proach  was  inadequate.  One  result  was 
the  finding  that  a  finer  grained  approacltto  representing  individual  userknoudedge 
is  required,  one  that  can  also  support  updates  as  users’  expertise  changes. 

The  explanation  approach  is  comprehensive  and  supports  all  four  layers. 
An  overview  of  the  users’  decision-making  process,  firom  the  point  the  system 
makes  a  recommendation  until  they  decide  to  accept  or  reject  the  suggested  trans¬ 
formation  is  shown  in  a  decision  flow  chart  in  Figure  6-3.  The  user,  when  in¬ 
formed  that  LlSP-CRmc  recommends  a  change  to  his  or  her  program,  can  get  an 
explanation  for  that  advice,  or  bypass  it,  deciding  right  then  to  accept  or  reject  the 
suggestion.  In  the  scenario,  the  user  followed  different  paths  through  this  decision 
process  in  different  episodes;  except  that  the  user  does  not  activate  the  hypertext 
facilities  in  any  of  the  episodes.  If  he  had  used  the  mouse  to  select  either  cond  or 
{/in  the  text  of  the  explanation  in  Figure  4-8  the  descriptions  shown  in  Figure  6-1 
would  have  been  displayed  in  the  LISP-CRITTC  window. 


Figure  6-3:  User  Decision-Making  Process  in  USP-Crttic 


Explanations  use  the  minimal  tq^roach  and  access  the  user  model  to 
determine  what  to  explain.  If  users  request  more  detail,  the  system  positions  them 
at  an  appropriate  place  in  the  hypertext  information  space;  once  there  users  have 
direct  control  of  access  to  other  hypertext  nodes  to  obtain  additional  information. 
They  terminate  the  explanation  dialog  when  they  are  satisfied  that  they  know 
enough  to  decide  whether  to  accept  or  reject  the  critic’s  suggestion. 

When  USP-CRTTIC  is  invoked,  the  user’s  code  is  examined  for  ways  to 
simplify  it.  In  the  first  scenario,  the  system  recommended  that  cond  could  be 
replaced  by  the  1/  special  form  Figure  4-7.  Before  the  user  decided  whether  to 


126 

accept  or  reject  LlSP-CRmc’s  suggestion,  he  asked  for  an  explanation.  The  sys¬ 
tem  then  determined  what  was  required  in  order  to  understand  ihis  rule,  and  the 
aspects  of  that  knowledge  that  tl^  user  lacked. 

The  explanation  component  inteirogated  the  domain  model  and  was 
provided  a  list  of  concepts  underlying  the  cond-to-if-else  rule.  This  list  was 
generated  by  traversing  the  domain  model  beginning  at  the  node  representing  the 
cond-to-ifivle,  and  using  the  dependent-on  links  between  domain  model  objects  to 
accumulate  the  concept  set.  For  the  cond-to-if-else  rule  in  the  first  episode  of  the 
scenario,  (Figures  4-7  through  4-9)  traversal  of  the  domain  generated  a  concept  set 
of  13  items:  lists,  symbolic-expression,  evaluation,  tests,  variables,  conditionals, 
scope,  predicates,  lisp-atom,  arguments,  falselempty-listlnil,  truelnon-nil,  and 
functions. 

That  set  was  personalized  for  the  user  in  the  scenario,  to.  determine  the 
subset  of  concepts  to  actually  be  explained.  As  discussed  in  Ch^ter  3,  there  are 
three  levels  dl,  d2,  and  d3  at  which  a  user  can  understand  a  given  concept.  The 
current  in^lementation  cs^tures  users*  knowledge  in  terms  of  concepts  that  are 
well  known  to  the  user  (dl),  known  to  the  user  but  not  well  (d2),  and  unknown 
(dO).  For  the  concept  set,  the  user  model  indicated  (by  their  absence  from  the 
concepts-known  slot  in  the  user  model)  that  the  user  has  no  knowledge  (level  dO) 
of  six  of  them:  predicates,  conditionals,  tests,  evaluation,  symbolic-expression, 
and  lists.  It  indicated,  based  on  their  markings  in  the  concepts-known  slot,  some 
knowledge  (d2)  of  6  others:  truelnon-nil,  falselempty-listlnil,  arguments,  lisp- 
atom,  scope  and  variable;  and  good  knowledge  fievel  dl)  about  just  one, 
functions.  That  information  was  provided  to  the  explanation  component  in  three 
sublists,  one  each  for  dO,  d2,  and  dl. 
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simple  text  fragments,  differential  explanations,  and  graphical-based  explanations 
similar  to  those  provided  in  the  Kaestub  system  [Nieper  83;  Boecker,  Fischer, 
Nieper  86].  Because  the  current  text  associated  with  each  is  stored  using  the  Con¬ 
cordia  hypermedia  system,  grr^hics  can  be  integrated  easily.  Concordia  is  a  hy¬ 
permedia  development  and  presentation  system  available  on  the  Symbolics,  the 
Document  Examiner  uses  it. 

The  problems  with  current  explanation  systems  are  recognized;  most  ef¬ 
forts  to  improve  them  emulate  human-to-human  communication  and,  too  fre- 
quendy,  attempt  to  provide  a  complete  explanation  in  one-shot.  Theoretical  results 
in  rhetoric,  and  discourse-comprehension  together  with  onpirical  observations  of 
human-human  collaborative  problem  solving,  indicate  that  trying  to  emulate 
human-to-human  human  conversational  techniques  may  not  be  the  best  approach. 
This  ch2q)ter  has  described  the  analysis  behind  a-  proposed  approach  to 
explanation-giving  that  tries^  to  consider  the  constraints  of  the  computer  interface 
while  taking  advantage  of  capabilities  and  resources  already  available  in  the  com¬ 
putational  environment. 

The  suggested  approach  provides  four  layers  of  explanation  for  the  ad¬ 
vice  given  by  a  knowledge-based  system.  The  first  two  layers,  although  they  can 
help  users  understand,  are  not  explanations  in  the  strictest  sense;  they  are  detailed 
descriptions  of  what  was  recommended.  The  3rd  and  4th  layers  use  a  minimal- 
explanation  approach  to  clarify  the  recommendations  and  expose  the  user  to  the 
underlying  rationale  for  that  recommendation.  Minimalist  explanations  need  a 
user  model  to  tell  the  system  what  is  necessary  for  the  user  to  understand  a  domain 
entity.  The  highest  layer,  a  rich  hypertext  space,  allows  users  to  explore  details  or 
examine  concepts  which  they  still  do  not  understand.  The  user  model  is  central  to 
this  proposed  framework  and  fundamental  for  the  explanation  approach.  The  next 
chapter  describes  that  user  model. 
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Fuithennore,  within  each  sublist,  concepts  were  ordered  according  to  an 
implicit  hierarchy  within  the  dependent-on  links  in  die  domain  model.  The  ex¬ 
planation  component  can  ultimately  use  this  information  for  reasoning  about  how 
to  generate  an  explanation  for  the  user,  but  the  current  implementation,  using  a 
simplified  strategy  for  testing  purposes  only,  selects  the  first  three  concepts  in  that 
filtered  list  predicates,  conditionals,  and  tests.  Ideally,  the  user  finds  the  explana¬ 
tion  adequate;  but  other  concepts  fundamental  to  understanding,  or  related  to, 
these  concepts  are  shown  as  mouse-sensitive  objects  displayed  in  bold.  Selecting 
any  of  them  will  display  either  an  explanation  associated  with  that  object  in  the 
domain  model,  or  a  description  from  the  Symbolics  Document  Examiner  (e.g. 
Figure  6-1). 

The  explanation  system  will  not  attenqn  to  present  explanations  as 
though  they  were  generated  by  an  intelligent  agent,  but  rather  use  combinations  of 
straightforward,  concise,  piewritcen  sentences;  What  distinguishes  this  ^^oach 
from  most  systems  that  use  canned-text  is  the  role  played  by  the  user  model  in  the 
process  of  constructing  an  appropriate  explanation..  Each  part  of  the  explanation 
can  be  chosen  using  the  domain  model,  the  user  model,  and  the  explanation 
strategies.  The  present  implementation  is  an  interim  step  to.  determine  the  efficacy 
of  the  user  and  domain  model  incrementations;  it  is.far  from  complete  and  future 
work  should  investigate  how  better  to  determine  exactly  which  concepts  to  ex¬ 
plain,  and  how  to  link  descriptions  of  them  together  with  information  about  the 
USP-CRmc  rule  of  interest  in  a  coherent  discourse  stroctuie. 

6.6.  Summary 

A  number  of  approaches  to  structuring  explanations  could  make  use  of 
the  available  domain  and  user  model  information.  They  include:  prestoring 
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USER  MODELLING  COMPONENT 

This  chapter  describes  the  user  modelling  component  for  LISP-Critic. 
That  component  acquires  the  user  model  and  represents  the  knowledge  of  each 
user  in  an  object  oriented  structure;  it  provides  access  to  that  model,  and  insures  it 
is  persistent.  The  component  was  implemented  in  the  Common  LISP  Objects  Sys¬ 
tems  (CLOS).  Access  to  individual  models  is  provided  via  a  set  of  generic  inter¬ 
face  functions;  other  system  con^nents  know  which  functions  to  call  to  obtain 
whatever  information  from  the  user  modd  that  they  might  require.  The  user 
modelling  component  invokes  the  appropriate  methods,  to  actually  access  a.  user 
model’s  contents;  it  uses  the  domain  model  stmcture  to  insure  that  appropriate  in¬ 
formation  is  provided.  The  accprisition  subconq>onent  provides  direct,  methods 
based  on  episodes  from  the  user-computer  dialog,  and  indirect  methods  triggered 
by  changes  to  individual  user  models.  The  primary  purpose  for  which  this  model 
was  developed  is  support  for  explanation-giving. 

7.1.  Design  Approach 

The  design  objectives  for  the  user  modelling  component  derived  from  a 
goal  of  supporting  the  explanation-giving  framework  discussed  in  the  Chapter  6. 
The  specific  implementation  t^roaches  selected  to  achieve  these  objectives 
resulted  from  the  analysis  of  other  user  modelling  research,  plus  the  requirements 
and  framework  for  user  modelling  needed  to  support  cooperative  problem  solving 
that  were  presented  in  Chapter  3. 
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7.1.1.  Objectives 

The  user  modelling  component  design  had  to  provide  support  for 
explanation-giving,  accommodate  various  acquisition  techniques,  and  be  able  to 
represent  a  variety  of  information  about  the  user.  The  model  c£q)tures  users’ 
domain  knowledge  and  supports  implicit  updating.  An  object  oriented  approach 
was  selected  for  implementation  in  order  to  insure  that  the  model  is  extensible,  can 
be  ad^ted  to  accommodate  other  techniques  (such  as  stereotyping),  and  can  be 
easily  modified  to  represent  new  kinds  of  information  about  users  (such  as  their 
preferences).  The  object  oriented  approach  allows  new  methods  to  be  defined  on 
existing  slots  in  the  model  and  new  slots  to  be  added  to  the  model  class  definition 
if  necessary. 

The  component  supports  both  implicit  and  explicit  update.  In  this 
research  the  implicit  update  techniques  were  the  primary  focus  however,  the  func¬ 
tions  which  modify  the  content  of  the  user  model  are  general  methods:  designed,  ta 
support  other  acquisition  q^roaches  as  well. 

A  model  that  we  are  able  to  use  only  once,  or  during  just  a  single  pro- 
granuning  session,  is  not  acceptable;  it  needs  to  be  retained  between  sessions  and 
reused  the  next  time  a  user  accesses  the  system.  Methods  that  save  the  contents  of 
the  model  at  the  termination  of  a  user-system  dialog  and  start  with  that  model  the 
next  time  a  user  accesses  the  system  are  included  in  the  component. 

The  model  must  support  changes  in  users’  knowledge  over  time.  It  is  in 
this  sense  that  the  model  is  dynamic;  its  contents  change  as  a  user’s  knowledge 
inqnoves.  In  this  regard,  the  emphasis  in  developing  inference  methods  was  on 
improvements  in  users’  knowledge  in  the  domaiiL  ITie  problem  of  how  to  modify 
the  model  when  users  forget  something  they  once  knew  is  an  irrqx)rtant  issue  but 
was  not  addressed  at  this  point  in  the  research.  That  the  model  is  at  best  an  ap- 


proximation  of  the  user  implies  that  the  modelling  component  must  be  designed  to 
include  techniques  for  improving  on  that  approximation.  Whatever  information  is 
available  to  enhance  the  model  has  to  be  used  to  best  advantage.  Specifically, 
what  a  model  represents  about  any  specific  user  should  get  better  during  sub¬ 
sequent  interactions  between  the  system  and  that  user. 

Three  different  approaches  to  representing  the  users  of  LiSP-CRmC  were 
considered  and,  in  some  cases,  partially  implemented:  classification  methods, 
ftereotyping,  and  an  overlay  of  the  systems  domain  knowledge  (the  USP-CRmc 
rules).  Classification  categories  and  stereotypes  are  similar,  but  for  discussion 
purposes  they  are  considered  separately,  as  were  the  attempts  to  use  them. 

Initial  attempts  to  model  the  user  with  classification  mediods  in  support 
of  explanation-giving  [Frank,  Lynn,  Mastaglio  87]  met  with  only  limited  success. 
The  problem  with  classification  methods  were  two-fold.  The  canned-text  explana¬ 
tions  directed  at  a  particular  level  of  expertise  were  found  to  be  unsafisfactory 
during  user  testing.  They  were  often  too  basic  to  satisfy  the  user,  or  too  difficult, 
using  concepts  not  yet  understood  by  a  particular  user.  Part  of  the  problem  is  how 
to  classify  a  user  into  one  of  a  set  of  prespecified  categories.  The  ones  used 
(novice,  intermediate,  and  expert)  did  not  tqrpear  to  c^mire  individual  expertise  in 
a  satisfactory  manner.  The  second  problem  with  classification  methods  is  that 
they  are  not  fine  grained  enough  to  provide  adequate  fidelity  in  their  representation 
of  individuals.  This  problem  was  confirmed  by  an  informal  study  in  which  the 
group  of  local  IJSP  programmers,  generally  considered  to  be  the  experts,  were 
asked  to  respond  to  a  questionnaire  about  their  use  of,  preference  for,  and  opinion 
about  teaching  certain  language  constructs.  The  responses  varied  widely,  indicat¬ 
ing  a  significant  difference  in  what  these  experts  knew  and  preferred.  When  es¬ 
tablishing  the  expertise  categories  in  a  domain  one  has  to  face  the  same  problem  as 
determining  the  contents  of  tqjpropriate  stereotypes. 
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A  schema  for  model  acquisition  using  stereotypes  of  LISP  programmers 
was  developed  [Fischer,  Mastaglio,  Rieman  89].  It  was  based  on  Rich’s  iqjproach 
to  stereotyping  [Rich  79],  and  showed  promise  as  a  way  to  leverage  analysis  of  the 
content  of  users  programs  to  stereotype  them,  and  from  that  stereotype  infer  ad¬ 
ditional  characteristics  indirectly.  Part  of  this  work  was  a  study  in  which  human 
LISP  experts  were  provided  a  program  and  asked  to  assess  the  expertise  of  the 
programmer.  Protocols  observations  in  this  study  showed  that  the  human  experts 
either  looked  for  or  noticed  what  we  called  “cues”  in  the  code;  cues  triggered 
inferences  about  the  expertise  level  of  programmers  who  wrote  them.  This  idea  of 
identifying  cues  in  the  context  of  a  user’s  work  is  something  that  was  carried  into 
the  acquisition  methods  finally  in^lemented^  The  methods  were  developed  and 
partially  implemented  but  this  Line  of  research  had  to  deal  with  the  problems  of 
what  stereotypes  to  use,  where  they  come  from,  and  how  to  insure  their  validity. 

Representing  a  user's  knowledge  as  an  overlay  of  the  existing  rule  base 
was  also  considered.  It  was  found  to  be  useful  for  guiding  critiquing  (e.g.,  making 
it  more  efficient  by  enabling  or  disabling  rules).  However,  a  model  that  only  ctq)- 
tures  user  knowledge  in  terms  of  the  LlSP-CRmc  transformations  is  inadequate;  it 
cannot  provide  the  required  support  for  explanation-giving.  There  are  slots 
provided  in  the  model  for  representing  rule  level  information  about  a  user;  the  im¬ 
plementation  therefore  does  provide  such  an  overlay  of  the  rule  base  for  use  in 
situations  where  it  is  of  value. 

The  limitations  encountered  in  considering  these  other  approaches 
provided  a  key  objective  for  the  design  of  the  user  modelling  component,  to  imple¬ 
ment  a  model  that  represents  user  knowledge  of  the  domain  at  a  level  that  is  of  fine 
enough  granularity  to  support  the  explanation  of  domain  entities.  The  basis  for 
that  representation  turned  out  to  be  the  same  deep,  domain  model,  as  required  to 
accon^lish  explanation-giving. 
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7.1.2.  Implementation  Approaches 

An  object-oriented  representation  allows  the  model  representing  each  in¬ 
dividual  to  be  idiosyncratic  but  for  all  the  individual  models  to  confirm  to  a  com¬ 
mon  format.  This  requirement,  coupled  with  the  need  to  support  easy  access  and 
the  changing  of  separate  instances  of  the  entire  class  of  models,  led  to  selecting 
that  object  oriented  representation.  The  structure  of  the  individual  models  is 
defined  as  a  class,  and  communicating  with  the  instances  of  that  class  is  facilitated 
through  methods  defined  on  it  The  object  oriented  representation  can  also  support 
potentials  enhancements  to  the  overall  user  modelling  component.  The  actual  lan¬ 
guage  chosen  for  the  object-oriented  representation  is  the  Common  LISP  Objects 
Systems  (Clos). 

The  need  to  achieve  a  representation  of  users’  domain  knowledge  in  a 
more  abstract  or  conceptual  form  than  the  LISP-CRTIIC  rules  resulted  in  basing  the 
user  model  on  the  conceptual  domain  model  described  in  Chapter  5.  The  domain 
model  did  not  pre-exist,  rather  the  motivation,  in  part,  for  developing  it  was  to 
provide  support  for  representing  us^’  domain  knowledge.  The  research  process 
concurrendy  developed  both  the  user  modelling  ^>proach  and  the  domain  model. 

To  support  dynamic  update  without  explicidy  querying  the  user,  the  im- 
plementadon  makes  use  of  informadon  available  in  the  context  of  the  human- 
computer  dialog.  Dialog,  in  the  sense  used  here,  refers  to  any  action  that  occurs 
between  the  system  and  the  user.  The  idea  that  the  model  should  be  implicidy 
enhanced  based  on  the  dialog  led  to  an  analysis  of  the  content  of  these  interactions. 
This  work  views  the  dialog  as  consistirg  of  a  series  of  episodes.  From  a 
hypothedcal  scenario  of  a  user  interacting  with  a  completed  system,  the  following 
kinds  of  episodes  in  LlSP-CRmC  were  identified: 
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•  user  requests  and  receives  an  explanation  of  critic  suggestion 

•  user  decides  to  accept  (or  reject)  critic  suggestion 

•  user  accesses  additional  on-line  documentation  to  help  clarify  an  ex¬ 
planation 

•  user  infonns  LlSP-CRmC  to  disable  (enable)  a  rule 

•  user  adds  a  personal  comment  to  an  argumentation  database  about  the 
tqjplicability  or  usefulness  of  a  transfoimation  in  the  rule  base 

The  implementation  uses  what  takes  place  in  those  episodes  as  a  primary  source 
for  triggering  system  inferences  about  the  users’  knowledge.  One  basis  for  this 
iq)proach  is  the  fact  that  users  apply  tiieir  knowledge  in  constracting  their  ’’side” 
of  the  dialog,  therefore  the  actions  they  take  provide  evidence  about  what  they 
know.  Just  as  significant  is  that  when  users  act  as  mere  receivers  of  information 
there  are  cues  here  as  to  how  the  user’s  knowledge  should  be  changing.  Specifi¬ 
cally,  they  should  now  have  command  of  the  domain  concepic  explained  by  the 
system.  In  this  second  case,  users  ‘Team”  from  what  the  system  tells  them  —  this 
is  the  basis  for  the  some  of  tUpct  inference  methods  that  will  be  described  later. 

The  above  objectives  and  approach  guided  the  manner  in  which  the  user 
model  is  represented,  acquired,  and  accessed.  An  architectural  overview  of  the 
user  model  component  in  Figure  7-1  shows  the  separate  subconqx>nents  and  func¬ 
tions  of  USP-CTRinc’s  user  modelling  component;  it  corresponds  to  the  general 
architecture  developed  and  shown  in  Figure  3-2  and  is  an  internal  view  of  the  user- 
model  as  one  of  knowledge-based  component  in  the  overall  system  diagram  that 
was  shown  in  Figures  4-4  and  4-5.  The  representation  subcomponent  will  be  dis¬ 
cussed  next. 
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The  user  modelling  component  is  one  of  the  knowledge-based  components 
of  LlSP-CRmc.  Data  are  indicated  with  an  oval,  collections  of  processes 
with  rectangles,  data  flow  with  directed  arrows.  The  three  subcomponents 
are:  a  representation  in  object-oriented  form  (Clos),  acquisition 
methods,  and  access  methods.  Acquisition  methods  modify  the  represen¬ 
tation  —  information  flows  from  them  to  the  representation.  Access 
methods  extracts  information  from  the  model  —  iirfotmadon  flows  from 
the  representation. 

Figure  7-1:  User  Model  Component  for  LiSP-CRmc 


7.2.  User  Model  Representation 

The  representation  is  designed  to  c^ture  a  variety  of  information  about 
the  user.  An  example  instance  of  the  class  user  model  is  shown  in  Appendix  B;  it 
is  the  state  of  the  user  model  at  the  conclusion  of  the  three  dialog  episodes  in  the 
Chapter  4  scenario.  The  interesting  part  of  the  model  (with  respea  to  this  project) 
are  those  slots  that  represent  the  user’s  expertise  in  the  domain  of  LISP: 
rules-known,  fimctions-known,  and  concepts-known  slots.  Conceptually  these 
record  the  coloring  of  the  domain  model  graph  for  the  user.  An  approach  that  was 
considered  for  the  representation  was  an  overlay  of  the  domain  model;  the  oveilay 
representing  those  domain  entities  a  user  knows.  But  the  model  needs  to  also  cap¬ 
ture  the  levels  of  the  users’  knowledge  according  to  the  classification  framework 
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described  in  Chs^ter  3,  the  ^roach  implemented  is  to  model  the  user  as  a  color¬ 
ing  of  the  graph  representation  of  the  domain.  To  demonstrate  this  and  show  how 
that  coloring  changes  over  time  we  will  use  the  previous  interaction  scenario. 

According  to  the  conceptual  framework,  a  user’s  knowledge  about  a 
given  concept  can  be  categorized  into  one  of  three  levels:  dl,  d2,  and  d3.  That 
firamewoik  was  shown  in  Figure  3-1;  it  provides  a  useful  scheme  for  ^roximat- 
ing  the  expertise  levels  of  users.  For  this  research,  we  adapted  it  to  represent  user 
knowledge  of  a  programming  language.  For  LISP  the  regions  in  the  graph  are  in¬ 
terpreted  as  follows: 

D^:  The  subset  of  USP  functions  and  underlying  language  concepts  which  users 
knows  and  incorporate  into  their  {nrograms  regularly,  they  understand  these  quite 
well. 

D2:  The  subset  of  concepts  which  users  know  and  will  use,  but  only  occasionally. 
They  does  not  know  the  details  nor  perlups  even  the  specific  syntax  of  functions  in 
this  region  but  are  aware  of  their  existence  and  have  a  general  understanding  of  their 
purpose.  Users  might  refer  to  a  LISP  text,  on-line  documentation  (e.g.  Symbolics 
Document  Examiner),  or  consult  a  colleague  for  help  in  coding  functions  in  this 
class.  Concepts  in  this  class  are  less  well  understood  by  users  than  those  in  D|  but 
still  can  be  considered  a  part  of  their  active  knowledge. 

D3:  The  conceptual  model  of  LISP  held  by  a  user.  The  concepts  and  functions  that 
they  think  exist  in  the  language;  this  region  also  includes  misconceptions. 

D4:  The  domain  knowledge  of  USP. 

The  inference  methods  that  were  developed  are  only  able  to  recognize 
domain  knowledge  in  dl  and  d2,  so  a  simplified  scheme  was  used.  It  conceptually 
marks  entities  in  the  domain  as:  well  known  to  the  tiser  (dl),  known  to  the  user  but 
not  well  (d2),  and  unknown  (dO).  Entities  at  level  dO  are  not  explicitly  listed  in  the 
user  model  but  the  component  infers  they  are  unknown  to  a  user  by  their  absence 
firom  the  appropriate  slot.  There  is  another  condition  that  could  be  the  reason  that 
a  domain  entity  is  at  dO;  it  may  be  that  the  system  has  not  yet  encountered  any¬ 
thing  to  trigger  an  inference  about  the  level  of  knowledge  —  it  just  does  not  know 
how  well  the  user  knows  the  entity,  if  at  all.  Discriminating  between  these  two 
situations  could  be  accommodated  with  methods  that  test  or  query  the  user.  There 
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are  situation  in  which  this  distinction  would  probably  be  beneficial,  tutoring  for 
example,  but  for  explanation  generation,  the  processing  required  to  distinguish  be¬ 
tween  them  does  not  provide  enough  additional  information  to  make  it  worth  the 
effort,  and  the  present  implementation  treats  both  situations  identically. 

Summary  data  for  user  model  for  SCENARIO-USER 

Following  concepts  in  D1 

FUNCTIONS 

Following  concepts  in  D2 

INTERNAL-REPRESENTATION 

SIDE-EFFECTS 

CONS-CELL 

VARIABLES 

SCOPE 

USP-ATOM 

ARGUMENTS 

FALSE/EMPTY-LIST/NIL 

TRUE/NON-NIL 

Following  functions  in  D1 

Following  functions  in  D2 

Following  Icr-rules  in  D1 

Following  Icr-rules  in  D2 

Rules-flred  by  name  and  number  of  firings 
NIL 


This  is  the  state  of  the  user  model  at  the  beginning  of  the  scenario.  The 
concepts  came  from  user’s  self-ratings  on  the  initial  questionnaire. 

Figure  7-2;  Initial  User  Model 

Recall  that  the  user  model  is  conceptually  a  coloring  of  the  graphical 
domain  model,  that  the  graph  has  concept,  function,  and  LlSP-CRmc  rale  layers, 
and  that  determining  what  to  explain  to  a  user  involves  extracting  information 
from  that  user  model,  the  tqjpropriate  concepts  required  to  understand  a  transfor- 
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madon.  Figure  7-3  shows  the  initial  coloring  of  the  concept  layer  for  the  scenario- 
user.  Concepts  known  to  the  user  model  are  shaded  sq^propiiately,  depending  on 
whether  they  are  in  dl  or  d2.  Unshaded  concepts  are  at  level  dO.  An  equivalent 
representation  that  shows  the  domain  knowledge  slots  for  the  user  model  is  tex- 
tually  displayed  in  Figure  7-2. 

In  the  scenario  the  system  suggested  a  transformation  based  on  the 
cond-to~if-else  rule,  the  situation  shown  in  Figure  4-7.  A  traversal  of  the  domain 
model  graph  generated  the  13  items  for  explanation  discussed  in  Chapter  6.  It  is 
now  up  to  the  user  model  to  filter  that  set  to  provide  assistance  to  the  explanation 
component  about  what  should  be  explained  and  how.  Most  significant  is  how  well 
(at  what  level)  the  user  knows  each  concept  That  information  is  provided  to  the 
explanation  component  in  three  sublists,  one  for  each  level  (dO,  dl,  and  d2),  which 
are  then  used  to  determine  the  explanation  strategy  and  presentation  approach. 

During  the  dialog  episode,  the  user  was  satisfied  with  the  explanation 
and  accepted  this  suggestion.  The  content  of  this  dialog  episode  was  used  to  up¬ 
date  the  user  model.  The  cues  from  this  episode  which  are  important  to  the  user 
modelling  component  are:  the  receipt  of  explanations  about  certain  concepts  (e.g. 
conditionals,  predicates  and  tests)  and  the  user  deciding  to  accept  the  cond-to-if 
transformation.  Cues  triggered  direct  inferences  that  changed  the  user  model  and 
these  changes  in  turn  triggered  indirect  inferences  that  will  be  explained  in  the 
next  section.  A  portion  of  the  updated  model  is  shown  textually  in  Figure  7-4,  and 
its  associated  graph  coloring  for  the  concepts  layer  in  Figure  7-5.  The  system's 
design  incorporates  techiuques  that  recognize  that  a  model  has  been  constracted 
for  this  user  and  makes  use  of  that  version  of  in  subsequent  dialogs  between 
USP-CRTTIC  and  this  user. 

The  model  was  saved  between  sessions  and  reused  by  LISP-CRITIC  when 
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Summaiy  data  for  user  model  for  SCENARIO-USER 


Following  concepts  in  D1 
FUNCTIONS 

Following  concepts  in  D2 

SYMBOLIC-EXPRESSION 

LISTS 

EVALUATION 

TESTS 

CONDITIONALS 

PREDICATES 

INTERNAL-REPRESENTATION 

SIDE-EFFECTS 

CONS-CELL 

VARIABLES 

SCOPE 

USP-ATOM 

ARGUMENTS 

FALSE/EMPTY-UST/NIL 

TRUE/NON-NIL 


Fbllowing  functions  in  D1 

Following  functions  in  D2 
IF 

COND 

Following  Icr-niles  in  D1 

Following  Icr-niles  in  D2 
USER:.-COND-TO-IF-ELSE 

Rules-fiied  by  name  and  times  fixed 

USER;:COND-TO-IF-ELSE 
TTMES-FIRED  1 
TTMES-ACCEPTED  1 
TIMES-REJECrED  0 


The  contents  user  of  the  model  after  the  first  dialog^  episodes.  Changes  to 
the  content,  when  compared  to  Figure  7-2,  are  a  result  of  user  actions 
during  the  episode  triggering  infermce  methods  that  update  the  model. 

Figure  7-4:  User  Model  Contents  after  First  Dialog  ^isode 


the  user  next  requested  critiquing.  In  the  scenario,  that  next  episodes  occurred 
when  the  user  requested  LiSP-CRmc  to  look  over  the  code  for  function  test,  as 
shown  in  (Figure  4-10).  The  first  transformation  recommended  on  this  code  is 
based  on  a  rule  called  de-morgan,  the  user  requested  that  this  rule  be  explained. 
The  user  model  again  provided  the  information  to  guide  the  process  of  developing 
and  presenting  that  explanation. 

When  the  user  accepted  the  suggestion,  his  user  model  was  again  up¬ 
dated.  The  system  suggested  a  second  transformation  for  this  function,  one  based 
on  the  cond-erase-pred.t  rule;  it  was  explained  in  Figure  4-11  based  once  again  on 
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Graph  coloring  for  user’s  concept  knowledge  after  first  episode,  also  shown  in  Figure  7-4. 
Figure  7-5:  Coloring  of  Conceptual  Graph  for  User  Model  after  First  Dialog  Episode 
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infonnation  from  the  user  model  about  how  well  the  user  knows  the  underlying 
concepts  for  this  rule.  It  happens  in  this  case  that  die  concepts  incorporated  into 
the  explanation  belong  in  the  user’s  d2  level  because  none  of  the  concept  set  un- 
deilying  that  rule  were  unknown  (at  level  dO).  A  third  suggestion,  based  on  the 
cond-erase-t.ml  rule  triggered  and  the  user  accepted  it,  but  without  an  explanation 
because  the  prerequisite  concepts  are  similar  to  those  already  explained  for  the 
previous  rule.  The  user  model  was  dynamically  updated  throughout  the  dialog. 
Each  time  information  was  shown  or  a  decision  made,  inference  methods  trig¬ 
gered.  The  domain  knowledge  portion  of  the  user  model  at  the  conclusion  of  the 
scenario  is  shown  in  Figure  7-6;  its  associated  gr^h  coloring  for  the  concept  layer 
in  Figure  7-7.  The  final  internal  representation  for  the  instance  of  the  class  user 
model  that  represents  scenario-user  is  shown  in  Appendix  B. 

Each  individual’s  model  is  an  instance  of  the  class  user  model.  A  user’s 
knowledge  about  each  category  of  domain  object  is  c^nured  in  slots  in  that  model. 
For  example  one  slot  contains  the  LISP  functions  a  user  knows  as  well  as  their 
level  Other  slots  contain  personal  infonnation  and  data  about  the  user’s  back¬ 
ground.  The  information  and  data  slots  could  be  fiUed  during  initial  start-up  of 
LiSP-CRmC  for  a  particular  user  (i.e.,  the  first  time  it  is  ever  invoked  by  them)  by 
several  explicit  acquisition  techniques.  Specific  the  mediods  to  do  this  have  not  yet 
been  implemented.  Instead  this  research  concentrated  on  developing  the  im|^cit 
inference  methods  that  modify  the  content  of  those  domain  knowledge  slots  during 
the  course  of  using  of  the  system. 

73.  User  Model  Acquisition 

The  user  modelling  component  contains  a  collection  of  methods  that  in¬ 
fer  which  domain  concepts  belong  in  the  user’s  model,  and  the  level  of  that 


Summaiy  data  for  user  model  for  SCENARIO-USER 


Following  concepts  in  D1 
SYMBOUC-EXPRESSION 
EVALUATION 
TESTS 

INTERNAL-REPRESENTATION 

SIDE-EFFECTS 

VARIABLES 

SCOPE 

LISP-ATOM 

ARGUMENTS 

FALSE/EMPTY-UST/NIL 

TRUE/NON-NIL 

FUNCnONS 

Following  concepts  in  D2 
LOGICAL-FUNCnONS 
LISTS 

CONDITIONALS 

PREDICATES 

CONS-CELL 

Following  functions  in  D1 

Following  fimcdons  in  D2 

NULL 

NOT 

OR 

AND 

IF 

COND 


Following  Icr-iules  in  D1 

Following  Icr-niles  in  D2 

USER::COND-ERASE-T.NIL 

USER::COND-ERASE-PRED.T 

USER:DE-MORGAN 

USER::COND-TO-IF-ELSE 

Rnles-fiied  by  name  and  times  fired 

USER::COND-ERASE-TJ«L 
TIMES-FIRED  1 
TIMES-ACCEPTED  1 
TIMES-REJECrED  0 

USER::COND-ERASE-PRED.T 
TIMES-FIRED  1 
TIMES-ACCEPTED  1 
TIMES-REJECTED  0 

USERr-DE-MORGAN 
HMES-FIRED  1 
TIMES-ACCEPTED  1 
TIMES-REJECTED  0 

USER::COND-TO-IF-ELSE 
TIMES-FIRED  1 
TIMES-ACCEPTED  1 
nMES-REJECrED  0 


The  State  of  the  user  model  after  the  second  (final)  dialog  episode. 
Figure  7-6:  User  Model  Contents  after  Second  Dialog  Episode 


Graph  coloring  of  user’s  knowledge  at  the  end  of  the  scenario,  also  shown  in  Figure  7-6. 
Figure  7-7:  Coloring  of  Conceptual  Graph  for  User  Model  after  Second  Dialog  Episode 
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knowledge.  The  information  that  triggers  these  methods  is  passed  to  the  user 
model  by  other  system  components  using  interface  functions;  in  turn,  indirect 
methods  are  internally  triggered  by  those  changes. 

This  research  investigated  how  a  user  model  for  a  cooperative  problem 
solving  system  could  be  enhanced  incrementally  over  multiple  dialog  episodes 
using  implicit  methods.  To  review  the  point  made  in  Table  3-1,  one  finding  was 
that  the  system  can  use  two  different  classes  of  update  methods: 

•  Direct  methods  that  use  a  specific  dialog  item  to  trigger  singular 
changes  in  the  user  model  information.  They  make  changes  to  the 
user  model  as  a  direct  result  of  information  the  user  receives  from  the 
system,  or  of  an  action  the  user  takes. 

•  Indirect  methods  that  are  triggered  whenever  a  change  to  the  user 
model  contents  occurs,  they  cause  further  updates  to  the  model,  one 
might  view  these  as  internal  demons. 

When  a  dialog  episode  triggers  direct  methods  that  change  the  user 
model,  these  changes  in  turn  trigger  indirect  methods  that  also  change  die  user 
model.  Indirect  methods  are  implemented  as  after-methods  on  slots  in  the  user 
model;  this  insures  that  they  get  run  whenever  a.  slot  change  occurs.  An  example 
of  this  chain  of  inferences  happens  in  the  first  episode  in  the  scenario.  The  user 
received  an  explanation  of  the  transformation  that  included  a  descr^on  of  the 
concept  of  conditionals.  That  concept  is  marked  in  the  user  model  at  level  d2 
based  on  a  direct  inference  method  —  when  the  system  explains  a  domain  entity  it 
assumes  the  user  now  knows  that  entity.  An  indirect  inference  method  is  triggered 
because  of  the  change  to  the  concepts-known  slot  That  indirect  method  inter¬ 
rogates  the  domain  model  and  determines  that  a  prerequisite  piece  of  knowledge 
for  understanding  the  concept  conditional  is  symbolic-expression.  This  causes  a 


change  to  the  model  that  adds  symbolic  expression  to  concepts-known  slot  at  the 
62  level. 

7J3.1.  Direct  Methods 

The  direct  methods  are  an  adaptation  of  related  woik  on  implicit  user 
model  acquisition  in  dialog  advisory  systems  that  was  discussed  in  Chapter  3.  The 
implicit  acquisition  rules  developed  in  [Kass  88]  are  based  on  using  natural  lan¬ 
guage  dialogs.  Here  dialog  is  used  in  a  more  general  way.  As  previously 
described,  it  means  any  of  the  different  interaction  episodes  that  occur  between  a 
user  and  the  system.  Using  this  view  it  was  possible  to  develop  our  own  set  of 
direct  implicit  methods  by  modifying  the  implicature  rules,  these  methods  fall  into 
four  major  categories: 

•  techniques  based  on  user  decisions, 

•  methods  triggered  by  information  provided  to  the  user, 

•  those  triggered  by  optional  actions  on  the  part  of  the  user,^  and 

•  ones  activated  when  users  access  the  hypertext  information  space. 

These  categories  of  information  are  available  to  the  system  through  traddng  its 
dialogs  with  the  user.  Appendix  C  shows  the  implementation  code  and  associated 
descriptions  for  each  specific  type  of  direct  inference  method.  Here  we  will 
describe,  in  general  terms,  each  category  and  the  rationale  behind  them. 

Dialog  episodes,  between  the  user  and  LiSP-CRmc,  terminate  with  a 
user  decision  (unless  the  session  is  aborted)  to  accept  or  reject  the  critic’s  advice. 
For  either  decision  users  requires  the  same  type  of  knowledge  (or  level  of  under¬ 
standing.)  In  both  cases  the  system  makes  the  same  inference.  A  decision  to  ac¬ 
cept  a  suggestion  made  by  LlSP-CRinc  causes  the  system  to  mark  the  rule  behind 
the  transformation  as  known  to  the  user  at  level  d2.  A  user’s  decision  to  reject  a 
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rule  is  handled  similarly.  This  igiioies  the  situation  where  users  reject  a  transfor¬ 
mation  because  they  do  want  to  bother  with  it  and  just  go  on.  The  system  im¬ 
plementation  does  provide  a  method  to  abort  the  interaction  and  it  is  assumed  diat 
users  are  sophisticated  enough  to  use  that  command  appropriately.  If  die  system 
served  a  group  of  users  less  computer-knowledgeable  than  programmers,  that  as- 
sun^tion  could  be  called  into  question. 

In  the  dialog  the  system  presents  information  to  the  user  in  the  fonn  of 
explanations,  an  event  that  triggers  a  direct  inference  that  users  know  the  ex¬ 
plained  entity.  When  the  system  explains  a  LISP  concept,  the  level  for  that  concept 
in  the  user  model  is  marked  as  d2.  In  the  first  scenario  explanation,  this  is  how  the 
concept  conditionals  came  to  be  madced  d2.  If  a  concept  just  explained  was  al¬ 
ready  mariced  at  level  62  then  its  level  is  improved  to  dl.  In  the  second  explana¬ 
tion  episode  the  concepts  internal-representation  and  side-effects  migrated  to  the 
dl  level  in  this  manner.  Similar  direct  changes  occur  when  functions  or  rules  get 
explained. 

When  users  encounter  explanations  that  are  unsatisfactory  they  can  ac¬ 
cess  a  hypertext  information  space  as  a  fallback  technique.  Their  selection  of  a 
mouse  sensitive  word  is  also  information  that  can  be  used  to  update  the  user 
model.  The  system  can  capture  the  selections  and  relate  them  to  domain.  modeL 
entities  where  possible.  This  csqsability  was  implemented  but  has  not  yet  been 
tested  to  determine  how  often  the  mouse  sensitive  objects  selected  match  the  en¬ 
tities  (functions  or  concept^'  in  our  domain  model.  When  a  match  is  found  the 
system  marks  the  domain  entity  at  level  d2  in  the  user  model  unless  it  is  already  at 
level  62,  in  which  case  its  level  is  improved  to  dl.  The  assumption  on  which  this 
method  is  based  is  an  optimistic  view  that  users  actually  read  and  understand  in¬ 
formation  provided  by  the  document  examiner;  an  assumption  that  could  be  sub- 
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jected  to  further  testing.  When  a  user  gets  an  explanation  of  the  selected  domain 
entity,  there  is  an  assumption  that  they  are  therefore  now  familiar  with  that  entity. 
Their  user  model  should  now  show  that  information.  The  domain  model  is  not 
necessarily  complete  and  there  are  may  be  inconsistencies  in  the  terms  used. 
However,  because  both  the  domain  model  and  the  document  examiner  generally 
use  accepted  terms  for  LISP  concepts  from  [Steele  84],  the  correlation  should  be 
high  enough  to  make  this  a  useful  approach. 

The  system  is  being  extended  to  allow  for  several  optional  actions  — 
actions  that  are  not  required  of  users,  but  allow  the  system’s  behavior  or  the 
documentation  base  to  be  modified. 

•  Users  can  change  the  action  taken  when  a  rule  fires;  they  can  tell  sys¬ 
tem  to  ignore  it  (always  reject  this  transformation)  or  automatically  to 
make  the  suggested  change  to  the  LISP  code  (always  accept  this  trans¬ 
formation.)  The  claim  here  is  that  users  must  understand  a  rule  before 
they  are  able  to  modify  what  happens  when  that  rule  fires.  It  is 
analogous  to  the  specific  decision  to  accept  or  reject  a  given  suggested 
transformation;  the  user  decides  to  accept  or  reject  it  in  all  cases. 

When  users  change  a  rule  status,  the  level  of  that  rule  is  caused  to  be 
set  to  d2  in  their  user  model. 

•  A  recent  extension  to  the  system  allows  users  to  associate  personal 
conunents  with  any  rule  in  the  documentation  space.  An  example 
might  be  a  programmer  who  docs  not  like  the  cond-fo-i/rule  because 
“it  creates  code  that  is  less  general.”  This  argumentation  can  be  at¬ 
tached  to  the  rule  and  associated  with  that  programmer.  These  com¬ 
ments  then  become  available  to  anyone  else  who  uses  the  system,  who 
can  also  add  their  own  comments,  perhaps  disagreeing  with  the  pre- 
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vious  author,  because  ‘'the  argument  misses  ^  point  that  the  rule  is 
intended  to  made  the  code  easier  for  other  progranuners  (ones  in¬ 
volved  in  maintaining  it  in  the  future)  to  imderstand.”  When  users 
involve  themselves  in  generating  such  argumentation,  the  system 
should  infer  that  they  understand  the  rule  quite  well  and  consequently 
mark  it  at  level  dl  in  the  user  model. 

Direct  methods  change  the  user  model  using  explicit  information  ob¬ 
served  in  the  dialog.  These  methods  alone  are  inadequate  for  developing  a  useful 
model  that  becomes  sufficiently  con:q>lete  in  a  reasonable  amoimt  of  time.  Ad¬ 
ditional  methods  that  leverage  this  information,  in  the  spirit  of  stereotypes,  were 
needed.  The  stracture  of  the.  domaia  model  provides  the.  basis  for  such  an  ad¬ 
ditional  class  of  methods  that  do  indirect  implicit  updating. 

7  J.2.  Indirect  Methods 

The  idea  for  indirect  methods  developed  while  implementing  the  con¬ 
ceptual  domain  model  when  it  was  observed  that  the^  links  in  the  model  csf>turing 
prerequisite  knowledge  for  the  domain  entities  could  provide  a  source  for  iiiq>licit 
acquisition.  These  prerequisite  were  established  for  use  in  explanation,  but  the 
idea  for  using  them  for  implicit  acquisition  resulted  from  noting  that  they  may  tell 
us  something  about  what  the  individual  knows  about  the  domain  —  in  the  spirit  of 
the  notion,  used  in  the  UMFE  system,  that  user  knowledge  “propagates”  through  a 
set  of  concepts.  The  pieiequisite  relationships  indicate  that  if  users  knows  a  given 
concept,  they  probably  know  its  dependent-on  concepts.  Based  on  this  obser¬ 
vation,  there  are  methods  that  trigger  whenever  a  change  occurs  in  the  knowledge 
level  of  an  entity  in  a  user  model.  Therefore  the  indirect  methods  leverage  the 
domain  model  structure,  allowing  the  system  to  enrich  its  model  of  a  user  without 


waiting  for  explicit  evidence  about  each  domain  entity.  The  indirect  inq)licit 
methods  belong  to  a  class  of  model  building  techniques  that  includes  stereotypes 
and  the  short  cut  methods  methods  used  in  human-to-human  cooperative  problem 
solving. 

Models  of  communications  partners  are  based  on  more  than  the  direct 
evidence  provided  direcdy  from  dialog.  Conqjuters,  as  Suchman  pointed  out 
[Suchman  87],  do  not  have  access  to  the  rich  set  of  information  available  to 
another  human  partner,  therefore  this  research  looked  into  ways  to  accomplish  a 
similar  eiuichment  of  the  model  by  using  available  resources.  One  such  resource 
is  the  linkages  between  entities  represented  in  the  stmcture  of  the  domain  model, 
these  links  form  the  basis  for  the.  indirect  update  techniques  in  LLSP-CRmc  A 
change  occurring  in  the  representation  for  the  user’s  domain  knowledge  can  be 
used  to  trigger  further  changes  to  the  domain  model  based  on  the  prerequisite 
knowledge  for  the  entity  just  changed.  The  indirectmethods  infer  how  well  those 
prerequisites  domain  entities  are  known  and  put  that  information  into  the  ap¬ 
propriate  slot  in  the  user  model.  Indirect  methods  exist  for  each  class  of  domain 
entity:  LlSP-CRTTIC  rules,  LISP  functions  and  LISP  concepts.  Their  implemen¬ 
tation  is  shown  in  Appendix  C. 

There  is  a  set  of  functions  associated  with  each  LlSP-CRTTlC  rule,  these 
functions  are  each  linked  to  that  rule  via  fimctions-in-rule  relationship  in  the 
domain  model;  they  are  the  functions  used  in  either  the  left  hand  side  or  right  hand 
side  of  the  rule.  Often,  rules  are  also  based  upon  certain  LISP  concepts  in  the 
domain  model.  When  the  level  for  a  rule  is  changed  in  the  user  model,  indirect 
methods  modify  what  the  user  model  has  to  say  about  how  well  the  user  knows 
those  associated  functions  and  concepts.  If  the  rule  has  been  set  to  level  d2  our 
indirect  methods  infer  that  the  functions  in  that  rule  are  also  known  at  level  d2,  and 
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nothing  is  inferred  about  the  concepts  behind  the  rule.  When  the  level  of  a  rule  is 
set  to  dl,  both  the  functions  in  that  rule  and  the  concepts  on  which  it  depends  are 
set  to  level  d2  in  the  user  model.  A  case  could  be  made,  in  retrospect,  diat  a  dif¬ 
ferent  level  might  be  inferred  for  the  functions  contained  in  the  left  hand  side  of  a 
rule;  after  all,  programmers  actually  use  these  in  their  code,  or  perh^s  that  the 
functions  in  the  right  hand  side  are  not  yet  a  part  of  a  user’s  LISP  knowledge  and 
should  not  be  added  to  their  model. 

For  a  LlSP-CRmc  rule,  the  domain  model  provides  information  about 
the  functions  in  that  rule  and  further  traversal  of  the  model  beginning  with  those 
functions  provides  a  list  of  prerequisite  concepts.  When  a  direct  method  modifies 
the  rules-known  slot  in  a  model,  an  indirect  inference  is  triggered  by  an  after¬ 
method  on  that  slot.  For  example,  for  the  cond-to-if  rule  in  the  first  explanation 
dialog  in  the  scenario,  first  that  rule  is  added  to  the  appropriate  user  model  slot, 
then  the  indirect  methods  also  add  the  cond  and  i/ functions  to  the  user  model  and, 
in  turn,  the  concepts  functions  and  arguments. 

The  prerequisite  to  understanding  a  LISP  function  are  its  dependent-on 
domain  concepts.  When  the  level  at  which  a  function  is  known  is  set  to  d2,  its 
dependent-on  concepts  are  also  set  to  level  d2.  If  the  level  of  a  function  is  set  to 
dl,  those  concepts  are  also  set  to  dl. 

Concepts  themselves  are  linked  to  one  another  via  the  dependent-on 
relationship.  When  the  level  of  a  concept  is  changed  to  dl  in  a  user  model,  its 
prerequisites  in  that  user  model  are  also  set  to  dl.  In  the  situation  where  the  level 
of  a  concept  is  changed  to  d2  its  dependent-on  concepts  are  marked  at  dl  when 
they  were  previously  marked  d2  in  the  model.  If  those  dependent-on  concepts 
were  not  already  in  the  model  (conceptually  marked  dO)  then  they  are  added  to  it 
with  a  d2  marking. 
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The  domain  model  implementation  also  contains  related  links  that  iden¬ 
tify  functions,  concepts,  or  niles  that  are  similar  to  another  function,  concept,  or 
rule.  This  class  of  links  could  be  used  as  the  basis  for  a  class  of  weak  inference 
meth'^ds,  such  as  predicting  the  ease  with  which  a  new  entity  could  be  introduced 
to  the  user.  In  the  ciuient  implementation,  the  similarity  relationsh^s  were  not 
exploited  in  the  present  user  model  acquisition  methods.  This  possibility  is  men¬ 
tioned  here  to  show  how  the  theoretical  approach  of  using  the  domain  model  struc¬ 
ture  to  infer  useful  information  extends  beyond  the  techniques  that  were  actually 
specified  and  implemented. 

The  implemented  indirect  methods  were  designed  conservatively;  they 
are  neither  complete  nor  perfect.  Their  shortcomings,  and  indications  about  how 
to  improve  them  came  out  during  an  evaluation  that  is  discussed  discussed  in 
Chapter  8.  There  are  two  results  firom  this  research  of  general  interest  and  utility: 

•  It  is  possible  to  define  a  class  of  implicit  user  model  acquisition 
approaches  that  are  indirect.  These  are  leverage  techniques  that  use 
information  about  users  that  is  not  directly  observed,  but  is  derived 
from  other  knowledge  (knowledge  of  a  domain  model,  stereotypes,  or 
etc)  to  indirectly  eruich  the  models  of  those  users. 

•  We  can  use  the  deep  conceptual  domain  model  that  is  needed  for 
proper  explanation  as  a  source  for  a  set  of  such  indirea  implicit  user 
model  acquisition  methods. 

The  methodology  followed  provides  an  approach  that  can  be  used  for  developing 
user  model  acquisition  techniques  to  serve  other  situations. 

In  summary,  to  show  how  the  direct  and  indirect  acquisition  techniques 
work  together,  let  us  review  a  portion  of  the  scenario.  The  concept  tests  is  marked 
at  the  d2  level  in  the  ..  er  model  shown  in  Figure  7-5.  This  resulted  from  a  direct 
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inference  based  on  a  method  that  claims  when  a  domain  entity  is  explained  to  a 
user,  that  user  is  now  aware  of  its  existence  and  has  a  fundamental  understanding 
of  it  —  the  user  knows  of  the  concept  but  is  not  proficient  in  ^>plying  it  in  every 
circumstance.  The  domain  model  also  tells  us  that,  for  the  concept  tests,  a  prereq¬ 
uisite  (according  to  the  dependent-on  links  in  the  domain  model)  is  the  concept 
symbolic-expression.  Therefore,  an  indirect  inference  places  symbolic-expression 
at  level  d2  for  this  user.  Similar  direct  and  indirect  user  model  acquisition 
methods  fire  for  the  cond-to-if-else  rule,  its  underlying  functions,  and  its 
dependent-on  concepts.  More  domain  entities  in  the  user  model  in  Figures  7-4  and 
7-5  get  marked  at  the  d2  or  dl  level  as  the  result  of  indirect  implicit  methods  than 
as  a  result  of  direct  methods. 

7.4.  Access  to  the  User  Model 

In  developing  the  architecture  for  the  user  modelling  component  one  ob¬ 
jective  was  to  insure  that  other  system  components  can  easily  access,  the  model. 
Another  consideration  was  to  have  a  model  that  supports  modification  to  incor¬ 
porate  additional  information.  Significant  theoretical  issues  or  results  were  neither 
addressed  or  discovered  in  this  aspect  of  the  work,  but  played  a  role  in  deciding- to 
use  an  object-oriented  approach.  Access  methods  support  the  current  explanation 
component  framework  while  providing  access  to  information  likely  to  be  of  value 
for  other  purposes. 

The  interface  functions  support  the  explanation  strategies  described  in 
(Chapter  6.  The  user  model  can  be  queried  to  determine  which  of  a  set  of  domain 
objects  a  user  knows  or  does  not  know.  The  interface  functions  are  shown  in  Ap¬ 
pendix  D;  some  examples  are  ones  that  determine  how  well  a  user  knows  a  domain 
entity  (i.e.,  at  what  level),  and  whether  a  domain  object  was  previously  explained. 
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An  attempt  was  made  to  conjecture  the  additional  information  a  user 
model  might  be  asked  to  provide,  and  include  in  the  framework  functions  that 
might  be  needed  in  other  situations.  Slots  in  the  model  record  all  rules-fired 
during  previous  dialog  episodes  and  the  number  of  times  a  user  has  invoked  the 
critic.  Other  such  information  includes  the  user's  goals  and  previous  programming 
language  experience.  The  goal  can  be  acquired  by  explicit  query  of  users  during 
their  initial  session  with  LlSP-CRmC;  ciirrently  it  is  defaulted  to  “simphiying” 
code  to  make  the  program  easier  for  others  to  read  and  maintain.  Previous  pro¬ 
gramming  experience  in  other  languages  can  also  be  obtained  through  such  an  in¬ 
itial  information-gathering  session  or  interactive  questionnaire. 

The  system  allows  the  user  to  modify  the  manner  in  which  the  system 
presents  information  and  the  default  action  taken  when  a  rule  fires.  It  supports 
end-user-modifiability.  The  user  model  contains  slots  that  record  such  user 
preferences  and  make  them  available  to  other  system  components. 

A  number  of  access  requirements  are  internal  to  the  user  modelling  com¬ 
ponent  itself,  the  instance-slots  can  only  be  directly  updated  by  the  modelling 
component.  The  component  receives  information  from  other  components  about 
user  actions  or  explanations,  and  determines  how  to  use  that  information.  It 
decides  what  additions  or  modifications  to  make  to  the  user  model  and  calls  the 
internal  methods  to  make  them.  The  user  modelling  component  is  notified  when  a 
session  terminates  normally  (is  not  aborted)  and  a  set  of  cleanup  actions  invoked. 
These  functions  save  the  user  model’s  current  contents  in  a  file  so  that  information 
is  not  lost  when  the  user  logs  out  or  the  system  is  rebooted,  and  can  be  used  during 
subsequent  log-ins  when  liSP-CRTIIC  gets  invoked. 

When  USP-CRmc  is  called,  the  system  determines  whether  the  user  has 
a  model  already  loaded  into  the  current  environment;  does  not,  but  the  system 
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saved  one  during  a  previous  session;  or  have  not  previously  used  LiSP-CRmc.  In 
the  first  case,  the  system  does  nothing;  in  the  second  case,  it  loads  the  most  recent 
version  of  the  user  model;  and  in  the  third  case  it  must  initialize  a  model  for  this 
programmer.  It  is  in  the  last  situation,  when  the  user  model  is  initialized,  that  the 
system  could  use  explicit  query  methods  to  gather  start-up  and  background  infor¬ 
mation  about  the  user.  No  explicit  methods  or  stait-up  user  questionnaire  were 
in^lemented  in  this  research;  such  an  in^lementation  would  not  add  to  the 
theoretical  ideas  developed  here. 

7.5.  Summary 

Specification  and  implementation  of  the  user  model  component  was  a 
major  portion  of  this  research  effort.  The  design  objectives  were  established  with 
a  goal  in  mind  of  generating  a  framework  ultimately  able  to  support  explanation  in 
any  cooperative  problem  solving  system.  Based  on  these  objectives,  specific  im¬ 
plementation  decisions  were  made  for  the  user  modelling  component  architecture 
for  LISP-CrittC.  That  component  uses  an  object-oriented  representation  scheme 
for  the  user  model,  a  set  of  access  methods  implemented  as  generic  interface  func¬ 
tions  that  can  be  called  by  other  components  to  interrogate  the  user  model,  and  a 
set  of  implicit  acquisition  methods.  The  latter  are  separated  into  direct  methods 
that  use  specific  information  to  trigger  an  inference  about  the  user’s  domain 
knowledge  and  indirect  methods  that  percolate  changes  through  the  user  model 
based  on  relationships  between  domain  entities  captured  in  the  domain  model 
gr^h  stmcture. 

The  outcome  of  this  theoretical  development  and  implementation  is  a 
framework  of  user  model  acquisition  techniques.  That  framework  can  be  used  to 
analyze  and  design  user  modelling  components.  The  update  methods  implemented 
here  were  the  subject  of  an  evaluation  that  will  discussed  next. 


CHAPTER  Vm 


EVALUATION  OF  THE  USER  MODEL 


8.1.  Introduction 

The  user  model  developed  in  this  research  project  was  evaluated  in  two 
ways.  Programs  written  by  students  learning  LISP  were  processed  using 
USP-CRmc,  and  the  individual  user  models  of  each  programmer  saved.  The 
models  were  compared  to  one  another,  attending  particularly  to  the  changes  that 
took  place  in  them  over  time.  Comparisons  of  the  contents  of  the  models  at  dif¬ 
ferent  times  permitted  an  evaluation  of  the  behavior  of  the  user  modelling  system, 
and  indicated  potential  system  improvements^  The  models  were  also  con^ared  to 
questionnaires  that  users  conq^leted  prior  to  each  of  the  three  programming  assign¬ 
ments.  The  data  collection  process  is  first  described,  then  the  results  of  analyzing 
those  data;  finally  I  discuss  what  these  results  inq>ly  regardingr  modifications  and 
enhancements  to  the  system. 

This  particular  evaluation  was  not  a  usability  study;  it  did  not  attenq)t  to 
assess  either  the  overall  effectiveness  of  LlSP-CRmC  nor  the  ability  of  program¬ 
mers  to  use  it  effectively.  These  types  of  studies,  as  discussed  briefly  in  Chapter  2, 
were  done  for  previous  systems  versions  and  helped  determine  the  capabilities  we 
want  to  provide  in  the  current  system  under  development.  In  this  work  the  em¬ 
phasis  was  on  developing  an  approach  to  user  modelling;  therefore;  the  evaluation 
focused  on  the  effectiveness  of  the  user  modelling  componem.  To  insure  consis¬ 
tent  and  useful  test  results,  it  was  necessary  to  control  the  test  scenario  conditions 
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to  which  the  the  user  model  acquisition  subcomponent  was  subjected.  The  ciment 
version  of  LISP-CRTTIC  does  not  have  a  fully  operational  explanation  component 
based  on  the  framework  described  in  Chapter  6;  because  the  presentation  strategies 
have  not  been  fully  defined  or  implemented,  a  total  system  test  was  precluded. 
Informal  studies  in  which  other  researcher  were  asked  to  “test  out”  the  system 
were  conducted,  and  the  results  integrated  into  the  interface  design  during 
development.  That  user  feedback  guided  decisions  about  menu  options  and 
names,  the  type  of  explanation,  and  what  capabilities  should  be  provided  for  users 
who  want  to  modify  the  system. 

If  the  user  modelling  acquisition  methods  wodc  properly  then  the  con¬ 
tents  of  the  user  models  should  both  change  over  time  to  reflect  first,  improved 
representation  of  a  user,  and  second  changes  in  the  smdents’  knowledge  itself  be¬ 
cause  they  were  engaged  in  a  learning  process.  Controlling  for  or  separating-out 
these  two  phenomena  was  not  possible  under  die  scenario  in  which  this  evaluation 
was  conducted.  However,  over  time,  the  user  models  when  compared  to  earlier 
ones  should  reflect  a  richer  representation  of  the.  student’s  knowledge  state. 
Secondly,  in  spite  of  the  limitations  of  a  self-assessment  methodology,  there 
should  be  some  correlation  between  the  model  contents  and  the  actual  state  of  stu¬ 
dents’  knowledge. 

8.2.  Data  Collection 

Lisp  programs  written  by  undergraduate  computer  science  students  were 
collected  throughout  the  Spring  1989  Semester.  These  students  were  enrolled  in 
CS3202,  Introduction  to  Artificial  Intelligence,  a  survey  of  artificial  intelligence 
techniques  which  provides  an  introduction  to  programming  in  LISP.  Ten  students 
volimteered  to  particq)ate  in  the  study.  We  collected  the  programs  which  were 
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submitted  to  fulfill  three  class  assignments.  Classroom  lectures  on  LISP  preceded 
the  assignments;  the  lectures  introduced  LISP  syntax  and  functional  programming 
techniques. 

The  three  assignments  were  spread  over  the  course  of  the  semester  with 
approximately  three  weeks  between  due  dates.  The  total  code  for  all  three  assign¬ 
ments  averaged  about  250  lines  per  student,  including  comments.  The  subjects 
also  completed  questionnaires,  an  example  of  which  is  shown  in  Appendix  E.  The 
questionnaires  accumulated  personal  and  experiential  background  as  related  to 
programming  and  asked  the  students  to  assess  their  own  knowledge  of  LISP  con¬ 
cepts  and  functions.  Three  questionnaires  were  administered,  one  before  each  of 
the  programming  assignments.  TI^  questionnaires  asked  the  student  to  rate  their 
knowledge  of  18  concepts  from  the  LISP  domain  model  (see  Chapter  5)  and  30 
LISP  functions.  The  rating  categories  were  designed  to  approximate  verbally  the 
levels  of  user  knowledge  that  were  discussed  in  Ch2^ter  3.  The  descriptive  rating 
categories  used  on  the  questionnaire  were: 

1.  the  student  could  define  the  concept  or  write  an  expression  using  the 
function  (dl), 

2.  for  a  concept,  this  rating  means  they  were  familiar  with,  but  could 
not  precisely  define  it;  and  for  functions  that  they  knew  of  its  exist¬ 
ence  but  would  have  a  problem  using  correct  syntax  (d2),  and 

3.  they  were  not  aware  of  the  concept  or  function  (dO). 

You  might  notice  that  it  was  possible,  on  the  questionnaire,  for  students  to  also 
classify  functions  into  a  category  indicating  that  they  had  heard  of  the  function  but 
were  not  entirely  sure  of  its  purpose  and  effects.  Functions  in  this  category  fall 
into  the  student’s  d3,  these  data  were  not  used  in  the  evaluation  because  the  user 
model  has  no  techniques  for  classifying  knowledge  of  domain  entities  at  that  level. 
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LiSP-CRmC  was  ran  on  each  student’s  programs  in  two  “test 
scenarios’’,  one  using  an  “accept’’  condition  and  the  other  using  an  “explain’’ 
condition.  Under  the  “accept”  condition,  the  simulated  response  to  each 
USP-CRmc  recommendation  was  to  accept  it  without  requesting  an  explanation, 
hi  the  “explain”  condition,  the  scenario  called  for  the  user  to  request  an  explana¬ 
tion  the  first  time  a  particular  LlSP-CRmc  rule  fired;  in  this  scenario  all  suggested 
changes  were  also  accepted.  Scenario  conditions  were  established  to  control  the 
conditions  so  as  to  limit  the  types  of  user  actions  to  which  the  user  model  acquisi¬ 
tion  subcomponent  was  exposed.  For  example,  access  of  the  hyptertext  documen¬ 
tation  space  was  not  called  for  in  any  test  scenarios. 

Programs  from  four  of  the  students  were  nm  through  LiSP-CRmC  under 
each  test  scenario  conditions.  These  four  were  selected  because  they  completed  all 
questionnaires  and  provided  completed  working  programs  for  all  three  assign¬ 
ments.  Six  user  models  for  each  student  were  captured;  a  set  was  saved  after  each 
one  of  their  three  programs  had  been  ran  through  a  scenario.  In  the  test  conditions, 
the  initial  (or  startup)  user  models  were  empty.  The  user  model  accumulated 
during  an  episode  under  one  of  the  scenario  conditions  was  retained  and  used  for 
the  succeeding  episode  under  that  same  condition.  For  example,  the  user  nsodel 
the  system  developed  for  userl  in  programming  assigiunent  one  under  the  accept 
condition  was  the  model  with  which  the  system  began  the  test  scenario  dialog  for 
userl  about  programming  assignment  two  under  the  accept  condition. 

8  J.  Analysis 

The  contents  of  the  user  models  were  analyzed  to  determine  the  total 
number  of  domain  objects  represented  at  knowledge  levels  dl  and  d2  after  each 
assignment.  Recall  that  any  domain  entity  not  explicitly  represented  in  die  user 
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model  is,  by  default,  considered  by  the  system  to  belong  in  dO.  The  results  are 
summarized  in  gn^hical  fonn  in  Figures  8-1, 8-2,  and  8-3. 

The  first  two  sets  of  graphs  (Figures  8-1,  8-2)  show  how  the  number  of 
objects  in  the  user  model  increase  over  time  (from  assignment  1  to  3).  Userl  un¬ 
der  the  accept  condition  scenario,  after  completing  the  first  assignment,  knew 
seven  LISP  fur^cdons,  according  to  his  user  model;  after  completing  assignment 
two,  eight  functions;  and  after  assignment  three,  ten  functions.  Under  the  accept 
condition,  domain  objects  never  get  ranked  higher  than  level  d2  because  collec¬ 
tively  the  acquisition  method  will  only  allow  a  domain  model  object  to  move  to 
level  dl  once  it  has  already  been  ranked  d2  and  either  is  explained  explicitly 
(which  of  course  never  happened  under  this  condition)  or  migrates  to  dl  because  it 
is  linked  via  the  dependent-on  relatious  to  another  domain  entity  that  moves  to  dl. 
Since  no  explanations  were  included  in  this  test  scenario  no  entities  ever  got 
nuu±ed  dl  to  begin  this  chain  of  inoferences.  Iris  therefore  impossible  for  any 
domain  entity  to  indirectly  migrate  to  the  dl  level.  A  similar  circumstance  exists 
for  LISP  functions  even  in  the  explain  scenario,  but  here  it  is  not  an  attribute  of  the 
control  conditions,  but  rather  indicates  a  possible  shortcoming  in  the  acquisition 
methods  that  will  be  discussed  later. 

Figure  8-3  shows  cumulative  results  for  all  three  types  of  objects  for 
these  four  students  over  the  three  assignments.  There  is  only  one  curve  for  the 
accept  condition,  the  total  for  the  number  of  entities  ranked  d2,  for  the  reason  dis¬ 
cussed  above.  The  shape  of  the  curves  in  this  graph  are  probably  what  would  be 
expected  from  programmers  learning  a  new  language.  They  initially  learn  a  few 
functions  and  concepts  to  get  them  familiar  with  the  language  and  able  to  write 
some  code,  and  after  some  experrence  they  begin  to  acquire  new  knowledge  at  a 
faster  rate.  This  is  the  type  of  learning  curve  one  would  expect  for  students  learn- 
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Figure  8-1;  User  Model  Test  Results 

These  graphs  show  changes  in  the  number  of  domain  entities  recorded  in 
the  user  models  of  two  students  under  the  two  different  test  conditions. 
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Figure  8-2:  User  Model  Test  Results 

Graphs  similar  to  the  ones  in  Figure  8-1  showing  changes  in  the  user 
models  for  two  more  students. 
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Summary  Data  for  All  Users  -  Both  Conditions 


Figure  8-3:  Summaiy  of  Test  Results 


ing  LISP  for  the  first  time.  It  rises  only  slightly  between  assignments  1  and  2,  then 
more  sharply  between  the  second  and  third  programming  assignments.  The  curve 
for  objects  being  ranked  at  level  dl  rises  less  sharply  overall  because  students  do 
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not  gain  complete  understanding  of  that  many  concepts  and  functions  over  the 
course  of  just  duee  programming  assignments,  but  they  are  likely  to  become 
quickly  familiar  Gevel  d2)  with  a  greater  number. 

Recall  that  the  semantics  of  domain  entity  ranked  at  level  d2  is  that  users 
know  about  such  entities,  but  would  need  assistance  in  jqrplying  them  in  their 
work.  This  situation  matches  both  the  course  objective,  to  introduce  LISP  and 
functional  programming  to  the  students;  and  reasonable  expectations,  students  do 
not  become  experts  after  three  programming  assignments  but  do  gain  a  more 
general  understanding  about  some  number  of  the  central  concepts  in  the  domain. 
In  general,  this  analysis  indicates  that  the  contents  of  the  tiser  models  correspond 
to  expectations  about  user  knowledge  under  the  conditions  set  for  the  evaluation. 
No  claim  is  made  diat  this  data  guarantees  that  the  model  representation  or  the 
acquisition  methods  are  valid;  instead  the  assessment  here  is  that  the  user  modell* 
ing  component  works  in  a  predictable  and  reasonable  fashion. 

8.4.  Results  of  Analysis 

The  analysis  provides  two  observations.  The  first  one  examines  the  ef¬ 
fectiveness  of  the  implemented  user  modelling  component;  the  other  one  considers 
the  accuracy  of  the  user  model  contents. 

8.4.1.  Efficacy  of  the  User  Model  Component 

As  discussed  in  the  scenario  presented  in  Chapter  7,  one  way  to  view  the 
content  of  the  user  model  is  as  a  coloring  of  the  conceptual  gnph  representation  of 
the  domain  model.  These  graph  colorings  together  with  data  discussed  above 
(shown  in  Figures  8-1,  8-2,  and  8-3),  point  out  some  potential  shortcomings  in  the 
user  model  acquisition  methods.  The  results  indicate  some  types  of  refinements 
diat  might  be  made  to  improve  the  inference  methods. 
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•  It  is  possible  for  the  acquisition  methods  to  infer  (color)  certain  ob¬ 
jects  from  the  domain  model  as  well  known  to  the  user  (level  dl)  even 
though  they  have  never  actually  been  explained  by  the  system.  This 
inference  is  perhaps  too  optimistic.  Certainly  some  users  generate 
self-explanations  for  some  concepts  or  rules  without  ever  consulting 
other  material,  but  there  is  no  guarantee  of  that  h^jpening  and  the  in¬ 
ference  methods  need  to  be  changed  to  wait  for  outside  confirmation 
of  that  knowledge. 

•  The  previous  situation  is  acceptable  under  some  conditions,  such  as 
after  observations  of  users  applying  the  given  concept  or  function  cor¬ 
rectly  in  a  subsequent  program.  The  a  priori  conditions  for  these 
types  of  indirect  inferences  should  be  made  more  stringent;  for  ex¬ 
ample  we  might  require  corroborating  evidence  from  other  sources, 
such  as  a  report  from  the  statistical  analysis  component  of  how-  fre¬ 
quently  a  programmer  uses  a  function. 

•  USP  functions  were  never  colored  at  the  dl  level  because  they  do  not 
get  explained  directly  under  the  present  strategy.  There  presently  are 
no  methods  to  infer  indirectly  that  a  user  knows  them  at  that  leveL 
This  is  partially  a  phenomenon  of  test  scenario  conditions  which  did 
not  call  for  using  the  hypertext  capability  as  a  fallback  technique.  Ac¬ 
tual  users  would  most  likely  have  used  that  facility,  for  example  call¬ 
ing  up  the  Document  Examiner  descriptions  shown  in  Figure  6-1  as  a 
fallback  to  the  explanation  for  the  cond-to-if  nle  shown  in  Figure  4-8. 
Again,  an  outside  source  could  provide  corroborating  evidence  show¬ 
ing  application  of  that  knowledge  (e.g.,  using  an  //in  a  follow-on  as¬ 
signment  in  the  situation  above). 
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8.4.2.  Comparison  of  the  User  Models  with  the  Questionnaires 

In  an  attempt  to  validate  the  inferred  user  models,  the  contents  of  those 
models  were  con^aied  to  the  responses  from  the  student  questionnaires.  Specifi¬ 
cally,  concepts  appearing  in  the  user  model  after  each  scenario  episode  were  com¬ 
pared  to  the  students  assessment  of  their  own  knowledge  of  those  concepts  at  the 
same  approximate  stage  of  learning. 

Table  8-1  shows,  for  all  users  and  as  a  total,  the  correlation  of  the  con¬ 
tents  of  the  models  after  processing  the  first  program  under  the  two  conditions, 
accept  and  explain,  with  the  second  questionnaire.  Similarly,  the  contents  of  the 
models  resulting  after  the  scenario  episode  for  assignment  two  were  correlated 
with  the  third  questionnaire.  The  questionnaires  were  administered  immediately 
before  the  students  received  their  programming  assignments.  The  second  ques¬ 
tionnaire,  completed  in  class  just  before  assignment  two  was  given,  therefore 
reflects  what  students  learned  about  LIFP  while  completing  assignment  one.  The 
classroom  lectures  on  LISP  were  formally  presented  at  the  beginning  of  the 
semester  while  students  worked  on  the  first  assignment,  and  should  not  effect 
these  correlations. 

Oi  Jy  correlations  for  concepts  were  computed  because,  as  it  turned  out, 
the  portion  of  the  questionnaire  dealing  with  functions  was  not  well  designed. 
There  was  minimal  overlq)  between  the  set  of  LISP  functions  on  the  questionnaire 
and  the  set  used  by  the  students  in  their  three  assignments  —  functions  acquired  by 
the  user  model  acquisition  methods.  The  questionnaire  was  developed  and  ad¬ 
ministered  before  getting  the  studenu’  programs  and  we  failed  to  include  many  of 
the  functions  they  actually  used  in  the  assigned  problems.  A  better  prior  analysis 
of  the  programming  problems,  and  conjecture  about  what  functions  might  be  used 
could  have  been  done;  the  functions  used  on  the  questionnaire  were  ones  that  we 
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Table  8-1:  Summaiy  of  Correlation  Results 


Coirdadoiu  of  User  Model  Contents  to  Qnestionn 

aires 

Condition 

User 

User  Model  1  vs  Quesdoonaiie  2 

User  Model  7 

>  vs  Questionnaire  3 

Accept 

Userl 

.25 

.15 

Usei2 

.08 

.17 

User3 

.18 

.18 

User4 

.92 

.23 

Total 

38 

.18 

Ejqilain 

Userl 

.67 

.77 

Usei2 

.92 

.83 

User3 

.45 

.36 

User4 

.23 

.69 

Total 

.56 

.67 

This  table  shows  correlations  of  the  user  models  contents  with  self- 
assessment  questionnaires.  When  the  scenario  call  for  users  to  receive  ex¬ 
planations  of  LlSP-CRmc  suggestions,  correlations  are  the  best,  greater 
than  50%  for  most  students  and  overall. 

thought  were  fundamental.  The  questiormaires  did  not  ask  about  all  of  the  45 
domain  concepts  and  103  functions  in  the  domain  model  because  of  a  desire  to 
keep  it  within  a  reasonable  length,  in  retrospect  it  might  have  been  better  to  ask 
about  all  of  them.  LiSP-CRmc  rules  were  not  asked  about  on  the  questionnaires 
because  they  would  not  have  had  any  meaning  to  the  students  in  an  abstract  form 
(e.g.,  by  name). 

The  correlations  under  the  explain  condition  are  significantly  higher. 
The  models  acquired  under  this  condition  correspond  better  to  the  self  assess¬ 
ments.  The  explain  scenario  is  probably  closer  to  the  process  student  program¬ 
mers  follow.  They  probably  engage  in  active  learning  while  in  the  process  of  do¬ 
ing  the  assignments.  They  encounter  and  learn  those  concepts  represented  in  the 
user  model,  and  on  the  questionnaire,  through  classroom  instraction,  or  by  consult¬ 
ing  additional  information  sources  (e.g.,  textbooks,  human  advisors,  or  peers). 
While  in  the  process  of  writing  their  programs,  they  seek  out  ‘'explanations”  that 
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help  them  to  build  a  mental  model  for  the  domain  comprised  of  the  concepts  and 
Lisp  functions.  Questionnaires  2  and  3  probably  should  nave  asked  the  respon¬ 
dents  the  types  of  source  materials  they  used  (e.g.,  textbook,  on-line  documen¬ 
tation,  etc.)  when  writing  their  LJSP  programs. 

8,5.  Limitations 

A  criticism  of  the  testing  process  is  that  because  the  students  themselves 
did  not  use  LiSP-CRrnC,  there  is  not  evidence  that  they  would  have  learned  the 
transformations  in  our  scenario.  Neither  do  we  know  if  students  actually  learned 
the  new  functions  in  the  right  hand  side  of  the  transfonnation  rales  or  the  concepts 
are  behind  them.  There  is  evidence  that  they  knew  some  of  the  functions  (the  ones 
in  the  left  hand  side  of  the  transformation  rules)  as  these  were  used  to  complete  the 
assignments.  The  evaluation  depends  on  the  assumption  that  the  subjects  became 
more  knowledgeable  in  the  domain  of  USP  because  they  completed  the  assign¬ 
ments,  and  on  the  assumption  that  the  code  used  in  the  test  scenarios  to  infer  how 
their  knowledge  changed  reflects  that  new  knowledge. 

The  domain  knowledge  required  for  any  single  transfonnation  requires 
understanding  the  functions  in,  and  concepts  underlying,  both  the  old  code  and  the 
new  code  from  a  transformation.  Therefore,  some  of  the  knowledge  attributed  to 
subjects  by  their  user  model,  came  from  code  the  students  wrote;  other  knowledge 
is  that  captured  in  the  LiSP-CRmc  rule  for  each  transformation.  In  principle,  noore 
than  half  of  what  the  system  infers  is  based  on  the  students’  code,  that  half  inferred 
from  the  left  hand  side  of  the  transformation  rules.  If  the  subjects  had  in  fact  ex¬ 
perienced  our  test  scenario  episodes,  one  could  hypothesize  that  the  rule  firings 
(and  explanations  if  requested)  would  have  caused  them  to  learn  new  functions 
and  concepts,  those  in  the  right  hand  side;  they  would  have  appeared  in  their  ques- 
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donnaire  responses  and  there  would  be  improved  correladon  between  the  user 
model  contents  and  the  quesdonnaire  data. 

8.6.  Shortcomings  in  System  Pointed  Out  by  the  Evaluation 

There  are  some  shortcomings  in  the  system  pointed  out  by  the  evaluation 
and  some  pointed  out  by  the  scenario.  The  scenario  was  presented  in  Ch^ter  3  as 
a  vehicle  for  understanding  the  context  of  this  woik.  However,  because  it  is  based 
on  a  set  of  programs  produced  under  actual  circumstances,  it  provides  insight  into 
how  the  user  modelling  component  works  in  such  a  real  scenario,  and  provides 
another  way  to  assess  its  effectiveness.  The  explanation  strategies  require  ad¬ 
ditional  implementation  work..  The  user  model  and  the  LISP  domain  model,  can 
support  richer  explanation  strategies  than  simply  the  display  of  hypertext  descrip¬ 
tions  for  the  underlying  concepts.  For  example,  the  domain  model  also  links  re¬ 
lated  objects,  such  as  similar  LlSP-CRITlC  rules.  This  information  could  be  used 
for  other  types  of  explanation  strategies,  as  previously  discussed.  The  explanation 
component  could  consult  the  domain  model  for  the  set  related  domain  entities  and 
then  interrogate  the  user  model  to  determine  if  any  of  the  entities  in  this  set  are. 
known  to  the  user.  Given  this  knowledge,  the  explanation  component  could 
describe  the  new  entity  in  terms  of  its  differences  hrom,  and  similarities  to  an  al¬ 
ready  known  entity.  Examples  occur  both  at  the  rule  level,  now  that  the  reader 
here  knows  die  rule,  a  reasonable  strategy  for  explaining  the 

cond-to-when  rule  would  be  to  use  this  differential  approach;  in  a  similar  fashion 
two  related-concepts  or  functions  can  also  be  described. 

Another  observation  is  that  domain  objects  migrate  to  the  user’s  dl  level 
of  understanding  in  the  user  model  too  easily.  One  can  see  this  graphically  by 
conqiaring  the  graph  colorings  from  the  three  scenario  dialog  episodes  in  Figures 
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7-3,  7-5,  and  7-7.  The  user  model  acquisition  methods  should  be  changed,  con¬ 
straining  inferences  to  “percolating”  knowledge  to  the  dl  level  through  no  more 
than  one  level  of  dependent-on  links.  Another  indicated  modification  is  to  the 
indirect  acquisition  rule;  instead  of  marking  dependent-on  concepts  at  level  dl, 
when  the  base  concept  is  at  dl  use  a  weaker  condition  that  marks  them  only  at 
level  d2.  We  should  consider  modifying  any  method  that  allows  the  user  model  to 
indirectly  infer  a  piece  of  knowledge  is  well  known  (level  dl)  to  a  user. 

8.7.  Implications  for  System  Modifications  and  Further  Development 

Three  major  findings  resulted  from  the  evaluations.  The  user  model  im¬ 
plementation,  particularly  the  acquisition  methods,  can  be  refined.  Using  a  startup 
or  initial  user  model  is  likely  to  provide  a  more  accurate  evaluation,  and  the 
domain  model  itself  could  probably  be  iteratively  refined  using  analyses  of  ad- 
ditiotml  test  cases. 

Refining  the  user  model  acquisition  process  means  that  methods  should 
be  modified  to  sqrply  less  optimistic  inferences,  as  just  discussed  in  the  previous 
section,  and  that  additional  methods  should  be  added.  The  indirect  methods  need 
to  be  modified  so  that  LISP  objects  are  added  to  the  domain  model  at  no  better  than 
the  d2  level.  Presently  the  indirect  methods  can  cause  a  domain  model  entity  to  be 
ranked  at  level  dl  and  that  is  probably  too  optimistic  a  point  of  view.  These 
methods  should  be  modified,  and  the  test  data  rerun,  to  see  if  better  correlations 
result. 

Using  a  startup  model  rather  than  beginning  “from  scratch”  would  es¬ 
tablish  a  more  realistic  test  scenario.  One  approach  would  be  to  use  stereotyping 
or  classification  approaches  to  provide  information  about  users  that  is  likely  to  be 
true  even  if  not  provided  directly  in  initial  questioiuiaires,  or  later  through  implicit 
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methods.  Observations  of  cooperative  activities  between  humans  [Reeves 
90]  showed  that  people  apply  certain  ‘leverage”  techniques,  such  as  stereotypes 
or  explicit  questioning  of  their  parmers,  to  provide  an  initial  or  default  model  to 
guide  their  first  interactions  with  another  person.  Implementing  these  types  of  in¬ 
itial  modelling  techniques  was  not  part  of  this  work,  but  some  simple  techniques 
could  probably  be  used  to  establish  initial  models  which  the  implicit  inference 
methods  could  then  improve  upon  during  the  test  scenarios.  The  questionnaires 
already  completed  by  the  subjects  could  be  the  source  of  initial  information  for  a 
startup  model.  A  test  of  the  system  starting  with  models  initialized  from  those 
questionnaires  would  probably  provide  a  more  realistic  scenario  of  how  the  stu¬ 
dents’  knowledge  changed  during  the  programming  exercises. 

An  analysis  of  the  graph  colorings  for  domain  concepts  indicate  that 
some  groups  of  concepts  (a  grouping  being  indicated  by  the  oval  size)  are  mote 
frequently  colored  and  perhaps  more  fundamental.  They  migrate  to  level  dl  the 
most  quickly;  they  are  the  ones  the  model  claims  are  best  known  to  the  user.  A 
more  detailed  analysis  of  the  models  generated  for  a  larger  population  of  users 
might  provide  some  insight  into  how  to  refine  and  improve  the  domain  model  to 
more  accurately  portray  programmers’  mental  models  of  the  domain. 

8.8.  Summary 

Testing  the  implemented  user  modelling  component  demonstrates  that 
the  techniques  woik  jqTproximately  as  was  expected.  Some  parts  of  the  user  model 
component  can  be  made  to  more  accurately  predict  user  knowledge,  and  any  sub¬ 
sequent  evaluations  should  employ  a  startup  model  to  make  it  approximate  more 
closely  the  approaches  people  use  in  similar  cooperative  problem  solving  sima- 
tions. 
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To  fully  validate  a  model,  such  as  the  one  proposed  and  implemented 
here,  will  require  significant  additional  iterative  development  togedier  with  exten¬ 
sive  testing  of  a  complete  critiquing  system  on  actual  users.  No  matter  how  well 
the  user  modelling  component  works,  it  produces  only  an  ‘‘approximation”  of  a 
user’s  knowledge  state.  People  themselves  make  do  with  similar  approximations 
of  their  communication  partners.  A  complete  user  modelling  system  will  need  to 
use  a  range  of  comprehensive  acquisition  methods  that  include  multiple  tech¬ 
niques,  as  will  be  described  in  the  next  chapter.  These,  together  with  detailed 
domain  and,  peih^,  task  models,  are  needed  if  user  modelling  is  to  become  a 
mature  technology. 


CHAPTER  DC 


APPUCATIONS  FOR,  AND  EXTENSIONS  TO,  THE  WORK 

This  ctu^)ter  analyzes  the  contributions  of  this  work  in  a  larger  context 
of  research  problems  and  q^plication  systems.  Primary  focus  was  developing  a 
user  modelling  tqiproach  for  critiquing.  But,  the  research  area  of  user  modelling  is 
important  in  a  more  general  sense;  my  results  can  be  of  use  in  several  other 
paradigms:  advice  giving  systems  [Wahlster,  Kobsa  88],  intelligent  computer- 
aided  instruction  [Wenger  87],  and  human-computer  interaction  in  general 
[Murray  Benyon  89].  A  primary  contribution  of  this  research  is  a  conceptual 
frameworlc  for  approaches  to  acquiring  user  models;  a  firamework  that  can  also 
guide  future  research.  A  possible  extension  is  to  eq>ply  our  specific  model  to  sup¬ 
port  £^lications  odier  than  critiquing.  Other  critiquing  q^lications  could  benefit 
fiom  the  addition  of  a  user  modelling  component;  it  may  be  possible  to  develop 
such  a  component  in  fashioo  similar  to  the  approach  followed  in  this  dissertation. 
Lastly,  there  are  also  several  interesting  directions  in  which  this  approach  to  user 
modelling  can  be  continued  and  extended. 

9.1.  A  Framework  for  User  Model  Acquisition  Techniques 

In  the  course  of  these  investigations  a  con^nehensive  framework  for 
classifying  approaches  to  user  model  acquisition  was  developed.  It  is  a  framework 
that  integrates  conceptual  discussions  [Wenger  87]  togedier  with  ideas  put  forth  in 
research  attempting  to  identify  the  acquisition  requirements  in  a  general  user 
modelling  rchitecture  [Kass  88].  The  framework  contains  four  categories  of  ac- 
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quisition  techniques.  Each  category  is  a  collection  of  methods  which  may  or  may 
not  be  appropriate  in  a  specific  system;  this  depends  upon  the  domain  and  type  of 
triplication  that  the  user  model  will  support.  The  purpose  for  the  framework  is  to 
aid  system  developers  and  researchers  in  identifying  appropriate  techniques  for  a 
particular  triplication.  The  framework  can  also  help  us  to  categorize  research  ef¬ 
forts  and  identify  problems  worthy  of  further  investigation.  This  framework 
helped  guide  the  implementation  of  the  user  modelling  component  for 
USP-CRmC. 

9.1.1.  Background 

In  the  process  of  developing  the  user  modelling  component  for 
LisP-CrtitC,  we  investigated  a  diverse  set  of  acquisition  strategies,  but  there  was 
no  methodology  that  could  be  used  easily  to  correlate  them.  A  classification 
scheme  that  helped  to  organize  the  ideas,  and  to  understand  research  on  user  model 
acquisition  by  others,  was  developed.  One  significant  finding  during  this  process 
was  that  user  modelling  for  many  types  of  systems  and  applications  require 
abstract,  conceptual  domain  representations,  and  furthermore,  a  number  of  the 
techniques  in  the  specified  framework  depend  on  that  deep  domain  model. 

A  motivating  factor  in  this  work  was  the  intuition  that  considerable  in¬ 
formation  about  the  user  is  available  within  the  computational  environment,  and  a 
desire  to  explore  how  to  make  use  of  that  information  for  user  model  acquisition. 
It  was  specifically  observed  that  users  demonstrate  their  understanding  through  ac¬ 
tions  that  they  take  and  by  the  decisions  that  they  make.  Also  noted  was  that  their 
knowledge  is  enhanced  whenever  they  are  exposed  to  system  provided  information 
in  the  form  of  explanation  or  advice.  The  central  issue  is  how  to  use  that  infor¬ 
mation  in  the  acquisition  process.  At  a  general  level  I  was  interested  in 
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'‘evidence-based**  approaches.  Systems  have  available  evidence  about  what  users 
know,  they  need  methods  that  tell  them  how  use  that  evidence  to  infer  models  of 
those  users.  The  indirect  implicit  acquisition  developed  in  this  research  also  had  to 
fit  into  the  framework. 

The  acquisition  framework  was  also  motivated  by  what  was  observed  in 
human-to-human  cooperative  problem  solving,  specifically  those  situations  in 
which  one  person  has  a  greater  understanding  of  the  task  itself,  while  a  second  is 
more  of  a  domain  expert  —  knows  more  potential  solution  approaches.  This  role 
distribution  is  similar  to  the  one  between  users  and  knowledge-based  computer 
systems.  In  the  study  of  sales  agents  assisting  customers  [Reeves  90],  when  inter¬ 
viewed  the  experts  related  that  they  use  direct,  questioning-types  of  approaches  to 
acquiring  a  model  of  their  clients  and,  more  interestingly,  several  consciously 
recognized  short-cut  techniques  (and  others,  we  suspect,  that  are  not).  The  trig¬ 
gering  conditions  for  the  inferences  are  interesting;  they  ranged  from  physical 
characteristics  of  the  customer,  the  way  the  client  is  dressed,  to  cognitive  traits, 
how  conversant  they  were  in  expressing  the  problem,  and  even  to  the  local  wea¬ 
ther,  was  there  a  significant  winter  storm  likely  to  cause  certain  problems  for 
homeowners,  automobile  operators,  etc.  The  classification  fiamework  attempts  to 
account  for  as  many  of  these  observed  techniques  as  possible. 

There  are  four  classes  of  update  techniques:  explicit  acquisition,  tutor¬ 
ing,  statistical  techniques,  and  implicit  methods.  With  present  technology  it  is  un¬ 
likely  that  a  single  system  will  be  able  to  use  methods  in  every  class  but,  in  build¬ 
ing  a  specific  user  modelling  component,  this  framework  can  help  in  the  selection 
of  appropriate  and  feasible  approaches.  This  is  not  meant  to  contradict  the  long 
term  goal  of  a  comprehensive  system  which  uses  multiple  approaches,  but  rather 
accounts  for  what  is  possible  to  do  in  systems  in  the  immediate  future. 
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9.1.2.  Explicit  Acquisition  Methods 

Explicit  user  model  acquisition  methods  are  based  on  direct  query  of  the 
user.  They  acquire  specific  information  about  the  user  that  will  assist  in  deter¬ 
mining  an  initial  user  model.  In  some  cases  they  are  used  to  clarify  conflicting 
information  in  a  model,  in  others  to  add  information  that  is  missing  but  needed  by 
the  system.  Explicit  acquisition  techniques  can  be  used  in  conjunction  with 
stereotypes  to  constmct  the  initial  model  of  a  user. 

Three  explicit  acquisition  methods  are  a  piescrq>tive  set  of  questions 
prestored  in  the  system,  dynamic  selection  of  questions  for  the  user,  and  fiee-form 
descriptive  user  input.  In  the  first  approadi,  a  system  developer  determines  what 
the  information  is  needed  for  the  initial  models  then  specifies  and  prestores  ques¬ 
tions  to  ask  of  the  user.  The  answers  provide  direct  information  to  enter  into  the 
model,  for  example  which  LISP  fiinctionsthey  already  knc" ',  or  can  be  designed  to 
trigger  richer  inferences  that  are  represented  in  sets  of  rulw,$,  procedures,  or  a  deci¬ 
sion  table. 

A  similar  q)proach  dynamically  generates  the  questions.  It  is  possible  to 
use  a  decision  tree,  or  the  structure  of  the  domain,  especially  if  it  is  hierarchical,  in 
conjimction  with  previous  answers  to  select  a  minimal  set  of  queries  to  establish 
an  adequate  startup  model.  An  example  from  the  UNIX  operating  system  domain 
is  to  ask  users  if  they  know  the  diff  coirunand;  a  positive  answer  would  allow  the 
system  to  infer  they  have  command  of  concepts  like  the  UNIX  file  system,  types  of 
files,  and  that  they  probably  know  more  basic,  related  conunands,  like  cat.  Is,  and 
more.  This  type  of  approach  was  explored  in  related  work  where  we  explored 
building  an  initial  user  model  for  a  learning  environment,  a  learning  environment 


designed  to  assist  new  users  of  a  workstation  [Mastaglio,  Turnbull  87]. 

The  third  technique  was  used  in  the  stereotyping  research  conducted  by 
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[Rich  79].  Here  users  describe  themselves  (their  interests  in  the  case  of 
GRUNDY);  the  terms  they  use  are  compared  to  stereotypes  the  systems  knows,  if  a 
favorable  match  is  found  the  content  of  the  stereotype  becomes  die  default  con¬ 
tents  for  the  initiai  user  model. 

9.1.3.  Tutoring-based  Methods 

Tutoring-based  methods  use  instractional  episodes  as  an  information 
source  to  inform  the  user  model  contents.  An  assumption  here  is  that  after  in¬ 
dividuals  receive  mtoring  on  some  domain  aspect,  they  now  know  it,  and  their 
model  should  reflect  that  fact  Tutoring-based  methods  are  used,  in  intelligent 
tutoring  system,  to  infer  student  models,  possibly  in  conjunction  with  additional 
methods  which  observe  a  studoit  subsequordy  using  that  knowledge  correcdy.  It 
is  possible  to  use  these  same  techniques  to  build  a  user  model  that  is  able  to  ac¬ 
commodate  a  more  comprehensive  system,  one  that  includes  a  tutoring  and  other 
components.  Student  models  are  primarily  used  to  determine  knowledge  that  is 
“missing”,  which  the  system  can  then  teach;  critics  are  interested  in  offering  ex¬ 
planations  for  similar  missing  knowledge,  when  it  is  required  to  tmderstand  a 
'critique.  These  related  needs  indicate  that  there  is  a  possibility  to  share  both  the 
user  model  and  the  acquisition  methods. 

Some  acquisidon  methods  in  tutoring,  attenq^t  to  determine  the  parts  of  a 
domain  that  are  misunderstood,  these  are  the  bug  approaches.  Knowing  the 
student’s  bugs  is  of  significant  value  in  guiding  an  instructional  process  but  it  is  of 
limited  use  in  applications  more  general  than  tutoring,  such  as  critiquing 

An  ultimate  objective  of  some  research  in  hiunan-computer  interaction  is 
to  provide  a  comprehensive  knowledge-based  system,  with  multiple  components 
all  supporting  users  in  an  interactive  working  context  —  an  intelligent  support  sys- 
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tern  [Fischer  86];  one  that  can  support  them  with  advice,  help,  tutoring,  critiquing, 
etc.  The  components  of  such  a  system  should  be  able  to  share  a  common  user  and 
domain  model.  Tutoring  episodes  provide  an  important  source  of  information  that 
can  be  used  to  enrich  the  common  user  model;  this  is  information  that  is  useful  to 
other  system  components.  ^ 

9.1.4.  Statistical  Analysis  of  User's  Work 

Statistical  methods  can  provide  a  measure  of  the  sophistication  of  a 
user’s  knowledge.  An  analysis  of  work  produced  by  the  user  could  be  accumu¬ 
lated  and  reported  to  the  system.  The  reported  statistics  provide  the  system  trig¬ 
gers  for  inferences  about  the  sophistication  of  a  user’s  knowledge.  For  example, 
in  Lisp  the  type  of  functions  used  (destracdve  versus  cons-generating)  might 
provide  a  system  evidence  about  the  user’s  overall  expertise.  Acquisition  methods 
based  on  statistical  iq)proaches  can  apply  machine  learning  paradigms  such  as 
learning  by  example  [Fain-Lehman,  CaiboneU  87].  The  examples  used  in  die 
machine  learning  process  are  the  users’  work.  The  analysis  of  what  a  user 
produces  could  take  place  as  a  separate  off-line  system  activity  [Fischer  87b].  Al¬ 
ternatively,  it  may  be  best  accomplished,  in  some  situations,  by  interpreting 
cumulatively  observed  data  about  the  user  over  time,  like  in  the  ACTIVIST  system 
[Fischer,  Lemke,  Schwab  85].  Statistical  and  mathematical  techniques,  such  as 
Bayesian  inference  and  fuzzy  set  theory,  can  provide  theoretical  bases  for  methods 
in  this  class. 

In  the  domain  model  described  in  Cluqiter  5,  some  categories  of  con¬ 
cepts  (or  functions)  appear  to  require  more  sophisticated  domain  knowledge  on  the 
part  of  the  user.  An  analysis  of  code  could  inform  the  system  about  the 
programmer’s  usage,  by  category,  of  both  functions  and  concepts.  This  infor- 
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category  for  the  programmer,  an  initial  user  model.  In  the  KNOME  system,  double 
stereotypes  were  used  in  this  manner  [Chin  89],  one  set  of  stereotypes  represents 
canonical  users,  the  other  set  is  a  categorization  of  UNIX  concepts  and  commands 
that  is  similar  to  the  groups  in  the  LISP  domain  model. 

The  major  problems  for  developers  is  determining  which  statistics  are 
important,  and  how  to  use  them.  Accumulating  statistical  data  about  a  user  is  not 
difficult;  the  issue  is  the  inferences  to  make  with  those  data.  The  conditions  under 
which  these  statistical  methods  might  work  is  possibly  domain  and  application 
system  dependent;  they  may  not  conform  to  a  general  theory:  this  is  certainly  an 
open  research  question.  Extensive  studies  of  user  populations  for  specific  systems 
will  be  required.  The  results  of  the  data  collection  have  to  be  correlated  with 
known  characteristics  of  the  users  to  determine  how  to  best  use  specific  pieces  of 
statistical  information.  Statistical  methods  are  similar  to  the  implicit  acquisition 
methods,  discussed  next,  in  that  both  operate  without  specific  input  fi'om  the  usen 
they  make  use  of  information  that  is  already  available,  information  generated 
during  the  course  of  the  user’s  normal  worit.  Like  stereotyping,  they  require  prior 
arudysis  of  a  sample  user  population  to  determine  what  certain  analytic  results 
might  imply  about  any  user.  Only  then  can  those  results  be  used  as  a  triggering 
condition  for  an  inference  method. 

9.U.  Implicit  Acquisition 

Implicit  acquisition  methods  use  the  contents  of  user-system  interactions 
to  make  inference  about  users.  They  are  designed  to  avoid  having  to  si  jject  users 
explicit  methods  that  interrupt  their  work.  Implicit  acquisition  methods  fall  into 
into  two  subcategories  direct  and  indirect  methods. 
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Direct  methods:  The  direct  implicit  inference  methods  observe  or  note 
user  actions  that  are  part  of  the  ongoing  user-system  dialog,  and  then  use  that  ob¬ 
servation  to  add  to,  or  change  information  already  in  the  user  model.  The  set  of 
implicature  rules  developed  by  Kass  are  an  example  of  methods  in  this  class  [Kass, 
Finin  89].  Direct  methods  are  based  on  die  idea  that  user-computer  interaction  is  a 
dialog.  Depending  upon  the  specific  application,  these  dialogs  have  different 
goals  and  formats.  The  dialog  may  seek  to  achieve  a  shared  understanding  be¬ 
tween  the  system  and  the  user,  or  to  negotiate  a  common  goal:  advisory  type  sys¬ 
tems  are  a  canonical  exan^le  of  this.  The  human  and  the  computer  might  also 
seek  agreement  on  whether  a  certain  course  of  action  is  ^iprqpriate;  knowledge- 
based  decision  support  systems  are  an  architecture  for  achieving  this  type  of  col¬ 
laboration  [Turban,  Watkins  86]. 

When  users  communicate  widi  a  system  in  any  form,  ranging  from 
natural  language  to  menu,  selections,  there  is  information  within  the  context  of 
those  interactions  that  the  system  can  use  to  infer  their  user  models.  Conversely, 
when  the  system  explains  something  to  users,  that  information  should  now  be 
known,  and  it  can  be  added  to  their  user  models.  In  LlSP-CRmc  the  direct 
methods  use  the  acceptance  of  critic  suggestions  to  trigger  one  type  of  direct 
method,  and  the  request  for  and  receipt  of  an  explanation  to  trigger  anodier.  The 
degree  of  the  system’s  confidence  in  that  part  of  the  model,  and  the  way  it  deter¬ 
mines  how  well  a  user  ’’knows”  that  information  are  open  questions,  the  answers 
to  which  may  also  turn  out  to  be  domain  and  application  dependent. 

Indirect  methods:  Tlu  indirect  methods  operate  like  internal  demons; 
they  use  changes  to  a  user  model  to  trigger  further  changes.  In  general,  any  in¬ 
ference  method  that  adds  not-diiecdy-observed  information  to  the  user  model 
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work,  the  indirect  metiiods  depend  on  the  support  of  a  deep  domain  model.  An 
example  of  an  indirect  method  occurs  when  the  user  model  is  updated  to  include 
the  fact  that  a  user  knows  a  certain  aspect  of  the  domain:  indirect  methods  in 
LiSP-CRmc  use  that  change  to  the  user  model  to  infer  that  the  user  also  knows  the 
prerequisite  knowledge  for  the  aspect  just  added.  Consider  an  example  from 
another  domain  that  demonstrates  the  generality  of  this  idea,  the  domain  is  math¬ 
ematics  and  here  the  system  observes  a  student  summing  two  negative  numbers 
correctly.  From  this  observation,  an  inference  is  made  that  the  student  knows  how 
to  sum  negative  numbers.  This  direct  inference  changes  the  user  model,  which  in 
turn  triggers  other  changes.  Oitt  inference  might  be  that  the  student  knows  the 
concept  negative  numbers,  another  that  he  or  she  knows  the  concept  addition. 
That  information  can  now  be  added,  to  the  user  model  if  it  is  not  already  present. 

92.  Employing  the  Approach  in  Other  Applications 

A  potential  qrplication  of  this  research  is  to  use  the  approach  reported  on 
here  in  other  types  of  applications  besides  aitiquingr  Th&  concqrt-based  user 
model  developed  here  has  the  characteristics  required  to  support  tutoring,  advisory 
systems,  and  human-computer  interaction  systems  in  general. 

Intelligent  tutors  frequently  represent  their  students  in  terms  of  produc¬ 
tions  contained  in  a  system  role  base,  or  in  terms  of  their  misconceptions.  The 
concept-based  user  model  representation  provides  an  alternative  method  for  guid¬ 
ing  the  tutor;  the  user  model  can  provide  a  list  of  those  concepts  that  a  user  does 
not  know.  Concepts  that  the  tutor  can  now  focus  on  teaching.  Concepts  which  a 
user  already  knows  provide  a  source  for  selecting  pedagogical  strategies;  this  is 
similar  to  die  methodology  used  in  the  genetic  graph  approach.  Opportunities  to 
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teach  new  concepts  using  analogy,  generalizaticm,  or  specialization,  can  be 
selected  by  comparing  die  user  and  domain  models.  To  actually  accomplish  this 
will  require  the  tutor  to  have  greater  understanding  about  the  semantics  of  the 
domain  model  structure,  specifically  the  links  between  entities,  as  well  as 
knowledge  about  what  didactic  approaches  are  suitable  under  what  conditions. 

Advisory  systems  give  advice  and,  in  some  cases,  are  also  designed  to 
assist  users  with  understanding  the  rationale  for  that  advice.  User  modelling  com¬ 
ponents  in  advisory  systems  focus  on  supporting  die  giving  of  advice,  they  infer 
user  goals  and  plans  to  insure  that  it  is  appropriate.  In  a  financial  advising  system, 
advice  would  consist  of  suggesting  to  users  where  to  invest  their  money  (e.g.,  in 
mutual  funds  or  municipal  bonds).  A  cmicept-based  user  model  could  assist  the 
system  to  explain  such  advice  —  answer  questions  such  as  why  are  municqial 
bonds  a  good  investment  for  me  at  this  time.  A  user  model  that  is  able  to  support 
both  the  advice  giving  and  explaining  roles  in  advisory  systems  will  have  to  be 
more  general,  c^turing  both  situation  specific  conditions  for  users,  such  as  their 
goals,  and  their  domain  expertise. 

An  advisory  system  that  will  be  consulted  on  multiple  occasions  by  the 
same  user  is  a  better  candidate  for  such  a  comprehensive  model,  for  example  the 
financial  advisor  above,  than  a  system  designed  to  provide  one-shot  advice,  such 
as  one  that  suggests  which  train  to  take.  A  hypothetical  example  is  an  advisor  for 
Lisp  that  can,  in  the  spirit  of  the  Programmer’s  Apprentice,  suggest  software 
cliches  that  will  accomplish  a  specified  task  (e.g.,  print  an  item).  The  system  has 
to  know  about  the  code  in  a  program  library  to  make  an  appropriate  recommen¬ 
dation  and  have  the  ability  to  explain  that  code  when  asked.  A  concept-based  user 
model  could  inform  the  system  during  the  advising  phase  to  help  it  select  a  cliche 
the  user  is  mote  likely  to  understand  (e.g.,  one  using  print  instead  of  format ),  and 
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during  the  explanation  phase  to  help  it  formulate  an  explanation  in  a  manner 
similar  to  that  envisioned  for  LISP-CRTIIC. 

Research  efforts  in  human-computer  interaction  often  claim  that  an 
idiosyncratic  user  model  will  enhance  that  interaction  [Murray  88],  but  focused  in¬ 
vestigations  along  these  lines  are  not  reported  in  the  literature.  The  fimdamental 
issues  are  determining  what  informaticm  those  models  must  provide,  and  how  that 
information  will  be  used  by  the  system.  One  direction  is  to  use  a  noodelling  ap¬ 
proach,  like  the  one  developed  here  for  critiquing,  as  a  starting  point  for  inves¬ 
tigating  how  system  arhqjtivity  in  the  general  class  of  human-computer  interaction 
systems  can  be  supported  by  user  models  [Murray  Benyon  89]. 

93.  Support  for  Critiquing  in  Other  Domains 

In  Chapter  2  we  coveted  the  ^rplication  domains  for  which  critiquing 
system  have  been  developed;  it  is  intriguingly  diverse.  Enhancing  the  effective¬ 
ness  and  utility  of  critics  with  a  user  modelling  component  is  an  indicated  future 
research  direction  in  several  of  the  system  descriptions.  A  useful  application  of 
this  research  would  be  to  enhance  a  different  existing  critic  system  using  an  ap¬ 
proach  similar  to  the  one  followed  for  the  work  on  LlSP-CRlTIC. 

Design  environments  [Lemke  89]  include  a  critic  component  and  some 
use  hypertext  issue-based  information  systems  as  a  source  of  information  for  help¬ 
ing  designers  understand  a  critique,  as  well  as  to  precipitate  reflective  practice 
[Fischer,  McCall,  Morch  89a;  McCall,  Fischer,  Morch  89;  Fischer,  McCall, 
Morch  89b].  No  attempt  is  made  to  ad^t,  or  tailor,  the  information  to  the  in¬ 
dividual  designer,  but  rather  the  methodology  focuses  on  presenting  it  in  a  strac- 
tured  manner,  and  insuring  it  is  contextually  related  to  current  work.  It  would  be 
worth  investigating  whether  user  models  such  as  the  one  in  LlSP-CRmC,  can  be 
integrated  with  these  techniques. 
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In  a  more  general  sense,  systems  based  on  critiquing  have  been 
developed  to  support  domains  such  as  software  engineering  [Fickas,  Nagarajan 
88],  VLSI  design  [Steele  87],  and  decision  making  [Mill  88];  systems  that  support 
knowledge  workers  who  use  them  repeatedly.  Users  expect  to  leam  ftx>m  die 
critiques  to  perform  their  tasks  better,  this  means  that  there  exists  a  need  for  sys¬ 
tem  explanations;  this  is  an  argument  similar  to  the  one  given  as  motivation  for  the 
work  on  explanation-giving  in  LISP-Crtitc.  Critiquing  is  also  used  to  support 
medical  jqiplications,  and  several  of  these  research  efforts  suggest  that  having  a 
user  model  would  enhance  their  systems  [Langlotz,  Shortliffe  83;  Miller  86;  Ren- 
nels  87].  Application  of  the  user  modelling  methodology  followed  here  to  en¬ 
hance  some  existing  critics  would  serve  to  validate  and  refine  the  techniques;  it  is 
also  sure  to  provide  additional  insight  to  motivate  improved  theory. 

9.4.  Issues  Warranting  Further  Research 

The  other  potential  direction  for  future  research  is  to  enhance  the  work 
accon:^>lished  thus  far.  Some  ideas  were  previously  mentioned  in  the  context  of 
describing  the  approach,  the  implementation,^  and.the  evaluation^  One  such-area  is 
to  leam  how  to  use  statistical  information  that  can  be  obtained  from  a  computer 
analysis  of  users’  work  or  actions;  another  is  to  integrate  mtoring  with  cooperative 
problem  solving  systems  to  determine  more  specifically  what  is  needed  iu  a  model 
designed  to  serve  the  needs  of  both.  One  extension  along^  these  lines  might  be  to 
build  a  comprehensive  system  from  scratch,  or  to  integrate  two  such  already  exist¬ 
ing  systems.  Some  other  potential  research  directions,  not  previously  mentioned, 
are  investigating  how  to  make^  better  use  of  networked  computing  environments, 
enhancing  the  domain  modelling  approach,  and  sharing  the  user  model  between 
multiple  applications,  or  even  difierent  domains  with  shared  conceptual  spaces,  for 
example  different  programming  languages. 
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Distributed  User  Modelling.  Present  computing  environments  are  al¬ 
most  always  part  of  larger  networks.  Having  other  machines  available  on  that  net- 
woik  should  allow  us  consider  how  to  introduce  concurrency  into  the  user  modell¬ 
ing  process.  Domain  models  will  come  to  have  greater  fidelity  and  a  richer 
representation,  and  user  models  will,  likewise,  become  more  comprehensive;  this 
may  cause  them  to  tax  the  computational  power  in  a  single  workstatioih  Also  we 
should  consider  the  situation  where  users  run  applications  on  remote  machines, 
applications  that  might  benefit  fipom  access  to  their  user  model. 

One  direction  for  research  is  to  investigate  how  to  provide  access  to  the 
user  model  stored  on  a  ‘‘personal  workstation”  to  qrplications  running  at  remote 
sites.  If  our  goal  is  a  truly  conprehensive  and  complete  user  model,  of  use  to 
multiple  systems,  then  it  follows  that  they  should  share  a  single  version  of  that 
model.  Some  issues  that  must  be  considered  are  privacy,  concmxent  updating,  and 
simultaneous  access  by  more  than  one  remote  server.  Research  into  using  the  ap¬ 
proach  in  this  manner  could  begin  by  determining  which  of  the  problems  involved 
can  be  solved  using  techniques  already  developed  in  other  concurrent  systems 
research  and  identifying  any  new  ones  that  are  generated. 

A  more  futuristic  idea  is  to  consider  having  a  user  modelling  machine, 
either  virtual  or  actual;  one  dedicated  to  performing  implicit  user  model  acquisi¬ 
tion  in  parallel  with  other  applications  to  achieve  concurrency.  Similady,  a 
machine  could  be  dedicated  to  the  role  of  donuiin  model  server.  This  might  be  a 
particularly  useful  approach  for  providing  reasonable  access  to  the  large  models 
that  are  growing  out  of  research  into  representing  general  common  sense 
knowledge  [Lenat,  Prakash,  Shepherd  86]. 
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Generality  of  the  Domain  Model.  The  domain  model  was  developed 
specifically  for  LISP,  in  order  to  meet  explanation  and  user  modelling  needs. 
During  this  research  it  was  observed  that  our  domain  model  is  an  instance  of  what 
other  researchers  in  explanation-giving  have  called  “deep  domain  models” 
[Chandrasekaran,  Tanner,  Josephson  88].  It  may  be  able  to  provide  support  for 
more  general  applications.  The  research  issue  is,  can  it  usefully  serve  other  ap¬ 
plications  or  interaction  paradigms,  other  than  critiquing,  and  if  not,  can  it  be 
modified  in  some  way  to  accomplish  this? 

Predominandy,  past  computer-based  systems  for  instruction  have  been 
one  of  tiiree  types:  dtiU  and  practice  computer-aided  instraction  that  captures 
domain  and  pedagogical  knowledge  directly  in  the  course  material,  intelligent 
tutoring  approaches  that  ctq)ture  domain  and  pedagogical  knowledge  in  produc¬ 
tions  and  exercises,  and  simulation  systems  that  c^ture  domain  knowledge  in  the 
behavior  of  the  simulated  devices;  pedagogy  is  implicit  in  the  simulation  process. 
This  is  not  to  say  that  these  will  be  the  dominant  future  t^roaches,  in  fact  we 
happen  to  believe  critiquing  [Mastaglio  89]  will  replace  or  augment  all  them  in 
certain  situations;  these  three  are  just  historically  the  most  common.  The  concept- 
based  domain  model  allows  pedagogy  to  be  derived  from  a  traversal  of  its  struc¬ 
ture,  links  provide  the  pedagogy  for  teaching  new  concepts  from  already  familiar 
ones.  It  would  be  worth  investigating  if  the  domain  model  could  be  used  to  help 
direct  a  didactic  con^ter  agent,  such  as  a  coach,  that  knows  the  learning  objec¬ 
tives  and  has  available  to  it  a  set  of  exercises  or  simulations  that  are  linked  to 
domain  model  entities  and  to  pedagogical  knowledge. 

Sharing  the  User  Model.  An  ultimate  goal  of  some  work  in  human- 
computer  interaction  research  is  a  comprehensive  system  which  supports  multiple 
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interaction  approaches  through  mult^le  con^nents.  In  terms  of  user  modelling, 
it  would  be  ideal  if  the  model  could  be  shared  by  these  components,  such  as  a 
system  incorporating  a  critic  and  a  tutor  like  GRACE.  It  would  be  worth  inves¬ 
tigating  what  type  of  user  model  is  needed  to  support  a  larger  class  of  qrplications, 
and  if  the  approach  discussed  here  needs  to  be  modijQed  to  achieve  this  goal.  As  a 
simple  exarrq)le,  some  of  the  knowledge  of  LiSP  captured  in  our  user  model  (e.g., 
conditionals,  scope,  tests,  etc)  would  also  be  useful  to  a  critic  in  a  related  domain, 
such  as  one  for  another  programming  language.  Identifying  the  common  domain 
characteristics  to  c£^ture  in  such  a  Glared  model  warrants  further  investigation. 

9.5.  Summary 

This  chapter  indicated  how  this  specific  work  fits  into  a  broader  perspec¬ 
tive  of  user  modelling  and  related  research.  The  acquisition  framework  can  help 
developers  of  systems  that  wiU  contain  a  user  modelling  coir^nent,  and  it 
provides  a  guide  for  future  research.  The  methodology  used  in  this  project  has 
potential  for  use  in  other  applications,  and  to  support  critics  in  other  domains. 
What  is  required  to  achieve  a  system  that  a  user  will  perceive  as  meeting  our  goal 
of  being  a  cooperative  problem  solving  system  is  still  an  open  research  issue:  the 
work  done  here  provides  a  starting  point  for  individualizing  the  types  of  environ¬ 
ments  in  which  we  eventually  hope  to  find  these  systems.  One  thing  this  disser¬ 
tation  research  has  shown  is  that  user  modelling  is  complex,  perhaps  one  of  the 
mote  complex  qrplications  yet  encountered  in  applied  computer  science  and  artifi¬ 
cial  intelligence  research.  It  is  not  possible  to  provide  complete  approaches  in  a 
single  research  effort,  and  solutions  are  neither  singular  nor  simple.  This  does  not 
mean  die  effort  is  not  important  nor  should  it  be  abandoned;  personalized  com¬ 
puter  systems  that  ad^t  to  our  needs  are  able  to  give  and  explain  meaningful  ad- 
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vice,  and  can  intopret  our  actions  are  a  consistent  image  in  science  fiction  and 
futuristic  scenarios  studied  by  serious  researchers  [SkuUey  88].  User  modelling  is 
one  of  die  important  enabling  technologies  needed  to  reach  that  goal;  but  arriving 
at  a  common,  useful  theory  will  require  multiple  efforts  and  the  synthesis  of  results 
in  order  to  understand  all  the  cognitive  and  conqiutational  issues  involved. 


CHAPTER  X 


SUMMARY  AND  CONCLUSIONS 

This  chapter  is  a  summary  of  this  dissertation  and  identifies  its  major 
contributions.  The  general  scheme  of  this  research  was  to  study  user  modelling 
research  in  other  areas;  to  develop  an  understanding  of  what  is  required  for  a  user 
model  to  support  cooperative  problem  solving;  and,  from  those  analyses,  to  devel¬ 
op  an  approach  for  supporting  a  computer-based  critic.  A  user  model  that  meets 
those  requirements  was  implemented  for  LiSP-CRmC,  and  subjected  to  an  evalua¬ 
tion.  The  results  of  the  system  development  work  and  analysis  suggested  ideas 
about  the  generalizability  of  the  methodology,  and  indicated  possible  extensions  to 
the  sfjproach  as  well  as  directions  for  future  research. 

10.1.  Summary 

This  dissertation  research  was  accomplished  in  a  context  of  developing  a 
paradigm  for  cooperative  problem  solving,  and  in  the  context  of  knowledge-based 
systems  that  support  user  learning.  In  principle  all  cooperative  systems  should 
also  support  learning.  Users need  access  to  system-provided  explanations  in  order 
for  that  learning  to  take  place.  Furthermore,  those  explanations  should  be  tailored 
to  their  individual  expertise  in  the  application  domain.  This  need  for  individual¬ 
ized  explanations  motivates  a.  requirement  for  idiosyncratic  user  models.  These 
characteristics  of  knowledge-based  computers  systems,  that  they  support  col¬ 
laborative  human-computer  efibrt,  and  also,  that  they  provide  learning  oppor¬ 
tunities,  determine  the  general  requirements  for  the  user  modelling  approach. 
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What  it  means  for  a  system  to  be  cooperative,  and  the  theoretical  characteristics  of 
learning  environments  were  discussed  in  Chapter  1. 

Critiquing  is  one  way  to  use  computer  knowledge  bases  to  aid  users  in 
their  work  and  at  the  same  time  support  their  learning  needs.  Research  investiga¬ 
tions  into  critiquing  by  the  Human-Computer  Communications  Research  Group 
has  included  system  building  efforts,  the  integration  of  cognitive  and  design 
theories,  empirical  observations,  and  the  evaluation  of  prototypes.  That  collective 
experience  was  integrated  with  a  smdy  of  other  research  to  determine  the  theoreti¬ 
cal  foundations  and  characteristics  of  critiquing. 

Chapter  2  presented  those  theoretical  foundations  and  the  theory  behind 
the  present  critiquing  firameworic.  Critiquing  systems,  also  called  critics,  are  an 
alternative  to  traditional  experts  systems.  The  generality  of  the  approach  is 
demonstrated  by  a  study  of  the  literature  which  shows  that  critiquing  has  been  suc¬ 
cessfully  used  in  diverse  application  domains,  hi  order  to  enhance  current  critiqu¬ 
ing  approaches  so  that  these  systems  move  from  simple  “suggestors”  of  how  to 
improve  a  user’s  work,  to  ones  which  can  interact  with  them  in  a  collaborative 
style,  will  require  models  of  users.  These  models  will  help  systems  adapt  explana¬ 
tions  for  their  domain  knowledge  to  individual  users.  Critics  are  not  the  only  ap¬ 
proach  to  building  better  knowledge-based  systems,  but  a  growing  number  of  such 
systems  will  contain  a  critiquing  component.  Some  of  them  need  detailed  under¬ 
standing  of  users’  problems,  tasks  and  goals;  but  more  commonly  they  will  have 
limited  yet  helpful  ctq)abilities,  one  of  which  is  to  model  the  knowledge  of  in¬ 
dividual  users. 

A  general  approach  was  chosen  as  a  result  of  studying  related  research  in 
user  modelling.  Chapter  3  discussed  user  modelling  in  other  research  areas  and 
the  foundations  for  user  modelling  to  support  the  types  of  cooperative  systems  in 


191 

which  we  are  interested.  The  approach  includes  an  architecture  for  a  user  modell¬ 
ing  component  comprised  of  a  representation  scheme  for  the  models,  acquisition 
techniques,  and  methods  for  accessing  the  models.  An  analysis  of  reported  work 
on  student  models  to  support  Intelligent  Computer-aided  lostraction,  and  user 
models  for  advice-giving  dialog  systems  determined  that  both  areas  provide  some 
important  concepts  to  help  us  establish  the  foundations  for  the  models  we  want  to 
have.  A  user  modelling  approach  for  cooperative  problem  solving  can  use  ideas 
developed  in  this  other  work,  but  it  was  not  possible  to  find  an  tq}proach  from 
other  research  that  could  be  adapted  directly  to  meet  the  needs  of  collaborative 
systems.  Therefore,  the  methodology  followed  was  to  identify  the  requirements 
for  a  user  modelling  component  to  support  explanations  based  on  a  theoretical 
model  of  users’  expertise,  a  conceptual  model  of  the  domain,  the  need  to  acquire 
the  model  using  implicit  methods,  and  all  the  while  keeping  in  mind  a  goal  of 
generality.  The  resulting  conceptual  architecture  was  instantiated  in  a  specific  sys¬ 
tem. 

In  Chapter  4  LlSP-CRmc  was  described:  it  is  the  environment  in  which 
the  implementation  work  was  performed.  LlSP-CRmc  provides  a  suitable  context 
for  investigating  user  modelling  because,  in  the  past,  it  has  been  a  development 
platform  for  investigating  various  notions  of  how  knowledge-based  computer  sys¬ 
tems  can  be  better  designed  to  acconunodate  their  users.  Some  of  these  ideas  were 
knowledge  representation  and  application,  user  access  to  the  systems  actions,  and 
explanation  of  advice.  Integrating  a  user  modelling  component  to  support  ex¬ 
planation  giving  was  a  natural  extension  of  that  previous  work. 

A  domain  model  was  required  to  support  both  explanation-giving  and 
user  modelling.  It  links  the  system’s  operational  knowledge  (LISP-CRTTIC  rules)  to 
the  domain  knowledge  necessary  for  explanation-giving  and  representing  users’ 
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expertise.  Chapter  5  covers  the  analysis  of  LISP  that  determined  what  to  represent 
in  the  domain  model,  and  dien  selected  some  i^Tpropriate  techniques  for  achieving 
that  representation.  The  inq>lemented  domain  model  cultures  knowledge  of  LISP 
in  a  conceptual  structure.  From  our  analysis  of  the  domain  it  was  determined  that 
the  fundamental  domain  entities  are  LiSP  concepts,  USP  functions  and 
LiSP-CRmC  rules,  they  are  all  interconnected  via  semantic  reladonsh^.  Concep¬ 
tual  gr^h  notation  was  used  to  visualize  the  domain  structure,  and  the  domain 
model  in  LiSP-CRmc  was  implemented  using  the  Common  USP  Objects  System 
(Clos). 

Cooperative  knowledge-based  systems  take  advantage  of  the  different 
strengths  of  users  and  computer  systems.  Computers  are  potential  sources  of  ex¬ 
pert  domain  knowledge  and  can  be  used  to  make  suggestions;  their  role  must  also 
include  the  ability  to  explain  diose  suggestions.  Explanation  systems  often  fail 
because  they  are  based  on  implicit  assumptions  that  explaining  is  a  one-shot  affair, 
and  that  artificially  intelligent  systems  will  be  able  to  retrieve  or  produce  complete 
and  individualized  text  Another  approach  is  to  take  advantage  of  information  and 
present  computer  technology.  The  explanation  ^>proach  discussed  in  Chapter  6 
focuses  on  determining  which  concepts  to  explain  to  a  user  rather  than  on  choos¬ 
ing  a  prestored  explanation.  Executing  that  process  requires  a  domain  model  that 
can  provide  the  set  of  concepts  needed  for  a  given  explanation  situation,  and  a  user 
model  that  can  help  tailor  the  explanation  to  a  given  individual.  When  explanation 
follows  this  qrproach,  the  process  is  one  of  constructing,  rather  than  selecting,  in¬ 
formation  that  will  be  presented  to  a  user. 

The  qjproach  used  provides  four  layers  of  explanation  that  can  be  ac¬ 
commodated  in  USP-CRinC.  The  first  two  layers  are  not  explanations  in  the  stric¬ 
test  sense  but  rather  techniques  for  presenting  the  critic’s  advice  that  facilitate  user 
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understanding;  they  are  detailed  descriptions  of  what  the  system  suggests. 
Rhetoric  principles  and  discourse  comprehension  research  provide  foundations  for 
a  minimal  approach  that  make  up  the  3rd  layer.  Such  minimal  explanations  are 
guided  by  a  domain  and  user  model  that  provide,  to  the  system,  information  about 
what  needs  to  be  explained  in  order  for  a  user  to  understand  a  particular  domain 
entity.  The  highest  layer  is  a  rich  hypertext  information  space  that  provides  a 
fallback  C2q)ability  for  simations  in  which  users  need  more  details.  In  that  hyper¬ 
text  space,  users  can  investigate  LISP  functions  or  examine  concepts  that  they  still 
do  not  understand. 

The  user  modelling  component  developed  forLlSP-Critic  is  described  in 
Chapter  7;  it  represents  what  die  system  knows  about  each  user  in  an  object 
oriented  structure,  acquires  those  user  models,  provides  access  to  them  and  retains 
them  for  future  use.  The  user  model  is  also  innplemented  in  CLOS.  The  design 
objectives  were  based  on  what  is  required  to  support  explanation-giving;  these  ob¬ 
jectives  guided  specification  of  the  architecture  for  the  user  modelling  component. 

Representation  of  the  model  is  an  enhancement  of  overlay  modelling 
techniques.  The  approach  ctqitures  the  domain  entities  a  user  knows,,  a.  subset  of 
those  represented  in  the  domain  modeL  but  also  maiks  them  in  accordance  with 
how  well  they  are  known.  Conceptually,  there  is  a  coloring  of  the  domain  model 
graph  unique  to  each  individual. 

Access  to  the  individual  models  for  other  system  components  is 
provided  for  with  a  set  of  generic  interface  functions.  In  this  research,  access  for 
the  explanation  con^nent  has  been  emphasized;  but  we  have  attempted  to  make 
the  methods  general  so  that  current  system  components,  such  as^the  critiquing  en¬ 
gine,  or  even  new  components  that  are  added,  like  a  tutor,  can  use  them. 
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The  acquisition  subcomponent  contains  direct  methods  that  make  use  of 
episodes  in  the  user-computer  dialog.  These  are  * ‘evidence-based”  methods  that 
resulted  from  the  intuition  diat  the  interaction  context  contains  useful  information 
for  inferring  the  knowledge  state  of  a  user.  The  subcomponent  also  contains  in¬ 
direct  methods  that  are  triggered  by  changes  to  individual  user  models.  One  out¬ 
come  of  the  LlSP-CRmc  system  development  and  implementation  work  is  a 
framework  of  user  model  acquisition  techniques. 

The  development  of  the  implicit  methods  is  considered  one  of  die  sig¬ 
nificant  contributions  of  this  research;  an  evaluation  of  their  effectiveness  and  pos¬ 
sible  modifications  was  undertaken.  That  evaluation  process  is  covered  in  Chapter 
8.  The  acquisition  methods  were  evaluated  in  two  ways.  Programs  written  by 
students  learning  LISP  were  processed  by  LiSP-CRmC;  and  the  individual  user 
models  for  each  programmer  compared  with  one  another,  attending  particularly  to 
the  changes  that  took  place  over  time.  In  this  way  the  behavior  of  the  user  modell¬ 
ing  system  could  be  analyzed  to  determine  potential  modifications.  The  user 
models  developed  for  each  student  were  correlated  with  questionnaires  assessing 
the  students’  expertise  according  to  the  topology  of  the  domain  model;  the  ques¬ 
tionnaires  were  conqileted  before  each  programming  assignment. 

The  evaluation  demonstrated  that  the  models  conform  to  expectations 
about  how  user  knowledge  might  change  under  the  conditions  these  programs 
were  produced.  The  models  contents  were  modified  by  the  qiplication  of  the  ac¬ 
quisition  methods  in  a  manner  similar  to  what  was  expected:  the  models  became 
more  detailed  as  the  system  was  exposed  to  more  of  the  users’  work;  and  they 
c£q>tured  new  concepts  as  students  learned  them  during  the  course  of  completing 
three  programming  assignments.  The  evaluation  pointed  out  opportunities  for  im¬ 
proving  the  acquisition  methods.  It  also  resulted  in  the  observation  that  using  a 
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staitup  or  initial  model  would  probably  improve  the  models’  fidelity.  The 
availability  of  an  initial  model  had  been  an  underlying  asstimpdon  in  this  research. 
This  finding  confirmed  the  importance  of  having  that  model;  regardless  of  the  ef¬ 
fectiveness  of  implicit  methods,  there  is  a  need  for  explicit  acquisition  of  initial 
models  of  users. 

To  achieve  a  completely  operational  model  will  require  significant  ad¬ 
ditional  development  and  extensive  testing.  A  limitation  is  that  any  user  model  is 
at  best  an  ’‘approximation”  of  a  user’s  knowledge  state,  therefore,  it  will  be  dif¬ 
ficult  to  determine  when  an  acquisition  methodology  is  as  complete  as  possible. 
An  outcome  of  this  work  is  an  awareness  that  a  conq>rehensive  user  modelling 
system  will  be  extremely  complex;  the  problem  will  not  yield  to  singular  solutions 
or  simple  methods  alone.  Researdt  efibrts  to  date  have  tackled  only  a  parts  of  the 
problem,  usually  in  isolation.  A  complete  implementation  will  have  to  integrate 
multiple  techniques  (e.g.,  stereotyping,  explicit  questioning  and  implicit  acquisi¬ 
tion  methods)  with  detailed  domaiir  and  peihaps  task  models^  The  work  under¬ 
taken  here  focused  on  developing  a.  domain  model  and  implicit  acquisition  tech¬ 
niques. 

In  cluster  9,  I  tried  to  demonstrate  how  this  work  contributes  to  a 
broader  scope  of  research.  One  such  contribution  is  the  acquisition  framework;  it 
provides  a  pretheoretic  scheme  which  can  be  used  when  developing  user  modell¬ 
ing  components  for  human-computer  interaction  systems  in  general,  and  can  serve 
as  a  guide  for  future  research.  Ttw  development  methodology  followed  here  can 
be  used  for  developing  user  models  for  applications  odier  than  critiquing;  and  to 
extend  those  critics  already  developed  in  other  research.  In  this  research  the  ap¬ 
proach  used  was  specifically  developed  for  critiquing,  but  it  provides  a  starting 
point  for  individualizing  a  more  general  class  of  cooperative  problem  solving  sys- 
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terns.  There  are  possibilities  to  share  both  the  model  in  its  current  foim  with  other 
interaction  approaches,  like  advising  or  tutoring,  and  to  use  the  methodology  as  a 
guide  for  developing  new  models  to  serve  a  range  of  iqjplications,  critiquing  com¬ 
puter  programs  being  just  one  them. 

10.2.  Conclusions 

The  woik  in  this  dissertation  project  contributes  to  research  in  user 
modelling,  explanation-giving,  and  cooperative  knowledge-based  systems.  The 
use  of  a  common  deep  conceptual  domain  representation  for  both  explanation 
generation  and  user  modelling  is  unique.  Using  the  inherent  structure  of  that  deep 
domain  model  to  perform  implicit  acquisition  is  a  technique  that  enhances  a 
system’s  ability  to  build  up  more  complete  idiosyncratic  models  of  users,  and 
should  be  explored  for  other  domains,  and  using  different  relational  links  between 
the  domain  entities.  The  explanation  process  begins  with  a  single  piece  of 
procedural  system  knowledge,  e.g..a  rule  that  a  user  wants  described..  It  serves  as 
a  starting  point  to  extract  the  appropriate  domain  concepts;  these  are  filtered 
through  the  user  model  and  some.  of.  them  are  eventually  explained  to  the  user. 
This  approach  could  potentially  be  used  in  a  large  class  of  human-computer  inter¬ 
action  systems  but  depends  upon  dcnnain  and  user  models  to  inform  the  process. 
This  work  is  a  first  documented  implementation  of  a  model  of  users’  domain  ex¬ 
pertise  in  a  critiquing  system.  The  possibilities  and  limitations  uncovered  here  can 
aid  developers  of  other  computer-based  critics.  The  framework  for  user  modelling 
acquisition  methods  proved  useful  in  developing  a  specific  user  modelling  com¬ 
ponent,  and  it  can  be  used  to  guide  design  analysis  and  architectural  specification 
for  others. 
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It  would  be  piematuie  to  claim  that  a  general  theory  of  user  modelling  is 
forthcoming,  but  this  effort  has  provided  a  better  understanding  about  some  sig¬ 
nificant  aspects  of  such  a  theory.  Specifically,  we  now  know  more  about  what  is 
required  of  a  user  model  that  supports  explanation-giving;  the  sort  of  techniques  an 
interactive  system  can  use  to  implicitly  acquire  such  a  model;  and  how  a  concept- 
based  domain  model  can  serve  as  a  basis  for  user  model  representation,  and  at  the 
same  time  support  user  model  acquisition.  These  ideas  expose  a  new  range  of 
issues  and  directions  for  research  into  user  modelling  that  may  eventually  provide 
general  mediods  able  to  accommodate  a  broad  class  of  human-computer  inter¬ 
action  systems. 
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APPENDIX  A 


USER  MODELS  REFERENCED  IN  DISSERTATION 

This  is  a  table  of  the  different  user  modelling  systems  discussed  through¬ 
out  the  dissertation  that  provided  insight  or  ideas  for  this  work. 
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APPENDIX  B 


SAMPLE  USER  MODEL 


This  £9)pendix  shows 


the  final  user  model  at  the  conclusion  of  the 


scenario  presented  in  Cht^r  4: 


f<USCR-MOOEL  S423771S>  1«  an  ln«tanc«  of  ciaaa  *<Standaxd-Claaa  USBR-MODBL  263S74S37>: 


Tho  following  alota  havo 

3H0ir*1IHY-BKrrER? 

SBOW-eXPLAHArZON? 

SHOIf-NBW-COOE? 

SHOW-OLD-OODB? 

PEMt-RULES 

OEFJarLT-AOCEPT-ROLBS 

ROLES  *TURlieO-OF7 

R0LE3-BXPIAIHE0 

FtmCTZOHS-BXPIAINED 

COMCBPtS-EXPIAZNBD 

ROLES^FIRED 


ROLES  *1010)01 
FOMCTIONS  -KNOW) 
COHCEPTS-KNOWf 


:XH8fANCB 

NIL 

T 

HIL 

T 

HIL 

HIL 

HIL 

(COHX>-BRASB-PRED»T  DSMORIGAM  COHD-TO-XF-EXSB) 

HIL 

(LISTS  LISP-ATON  FALSE/BMPTy-LIST/NIL  TROE/HOH-KIL  TESTS 
CCmiTZOMALS  PREDICATES) 

((OOHD-BRASE-T.NIL  (TXMBS-PIREO  .  1)  (TI»S-ACCBPTED  .  1) 
{7ZME5-RBJBCTB0  .  0)) 

(COHD-BRASB-PRBD.T  (TZHBS-FIRED  .  1)  (TIMBS-ACCBPTBD  .  1) 
(TIMBS-RBJECTBD  .  0}) 

(DB-MORGAH  (TIMBS-FIRED  .  1)  (TIMBS-ACCBPTBD  .  1) 
(TZMB5-REJBCTBD  .  0)) 

(COHD-TO-IF-ELSB  (TIMBS-FIRED  .  3)  (TIMB8-A0CEPTED  .  3) 
(TIMES-REJECTED  .0))) 

((COHD-BRASB-T.HIL  .  D2)  (COHD-BRASE-PRED.T  .  D2) 
(DB-MORGAN  .  D2)  (COND-TO-IF-BLSE  .  Dl>  ) 

((ROLL  ,  02)  (ROT  .  D2)  (OR  .  D2)  (ADD  .  D2)  (IF  .  D2} 
(CORD  .  D2)) 

( (LOGICAL-FOMCTIORS  .  02)  (SIMBOLZC-EXPRE55IOH  .  01) 

(LISTS  .  Dl)  (EVALOATIOH  .  01)  (TESTS  .  01) 

(commoHALs  .  oi)  (predicates  .  oi) 

(ZRTBRRAL-REPRESBRTATION  .  02)  (SIDE-EFFECTS  .  02) 
fCORS-CELL  .  02)  (VARIABLES  .  01)  (SCOPE  .  01) 

(LISP-ATOM  .  01)  (ABCOMERTS  .  01)  (FALSB/EMPTY-LIST/NIL  . 
(TROB/ROH-NIL  .  01)  (FUNCTIONS  .  01)) 


PROGRAMaNG-LARGOAGE-EXPERZENCB 


01) 


USER-COAL 
TIMBS-LCR-INVORED 
DATB-LAST-OPDATED 
USER-HOME -OZRBCTORT 

NAME 


(PASCAL  C) 

SDVLIFT 

3 

•2/23/PO  13;52;27« 
#P*MORaS  :>SCERARXO-OSER>” 
SCBKARZ0-U5BR 


APPENDIX  C 


INFERENCE  METHODS  IN  USER  MODELLING  COMPONENT 

This  appendix  shows  some  of  the  code  that  implements  the  acquisition 
subcomponent  of  the  user  modelling  component  for  LiSP-CRmc.  The  im¬ 
plemented  methods  to  implicidy  infer  information  about  the  user’s  domain 
knowledge  are  shown  below. 


tnt  DIKBCT  HCnOM 

thm  iat«rf«o«  fuactiou  ttot  otter  AyatM  poll  to 

; ; ;  Lot  tte  upor  oodol  know  ttet  m  twos  teo  f  kri>  ooxtote  ootlooo  •  Tte.  f icot. 

ot  fttaotloo*  pso  tte  dlroot  Mttetei  tk—  lafoc  ttet  ooxtote 
tniAfostetlott  tteote  te  arttett  to  tte  «tec  ante!  ♦  Tteoo  faaotioao  oaa  te 
;;;Tlo«od  tte  laplwpwif  tioa  of  •  ■oaoayo  paosite  pcotoooX  botit— i  aotel— . 

(teCwi  TIZJ.-0«Slte30KL-ianJE-JU9eirTB>  IraKi-ri— > 
nit  •  teor  aoooptr  «  nlo  «•  lAfoc  ttet  tboy  oteorotMWl  ttet  rolo  at  tte  d3  lovoX 
<odd-to*toor"tedoX  *<nirroiit'-ajwxtedoX*  rtlifo 

;;«■  aXao  odd  tte  foot  ttet  ttte  r«Xo  tea  flrod  aa«l  woo  aeooptod  te  rtXoo^flrod  aXot  of  ooor  aodoX 
<tipdoto»>raX— "'flgod-aXet  ruXo-MMO 


<dOf«B  TSLL-OmMOKL-ROIX-RSJBCTO  Inn  TifT-? 

;iif  a  tear  rajaota  a  rtXa  wa  iafar  ttet  they  atearataad  ttet  naXa  at  tte  6Z  XaaaX 
(add  to-aaar"'aodal  *e«rxaat*aaaaMdaX*  rmXa*^M» 'd^ 

;;wa  aXao  add  tte  fact  ttet  ttia  raXa  tea  flsad  and  waa  rajaotad  to  raXaa~flrad  slot  of  uaar  aodaX 
(apdata-TuXaa-flrad-aXot  raXa-aaaa  'ra)aatadM 


(dafaa  TKUi*OSSM«X>KX<-caiPCBPT-KXrX«AXV»  (oeaoapt) 

;;aftac  a  ooaoapt  ia  ajylaiaad  aa  pXaoa  tbte  Baacipt  te  tte  aaar  aedaX  at 
;rXaaaX  df,  uaXaaa  it  ia  aXraady  praaaaty  te  ttet  oaaa-  aa  spqxada  Ita  XaaaX  to  dl 
;;wtea  tte  ooaoapt  ia  aXxaady  teoaa  at  XaaaX  dir  aa  do  aothtep 
(aoaa  (odr  (aaaoa  ooaoapt 

(ooaooptfkaoaa  *oarraat*aaafodaX*) ) ) 

(<ail)  (add"t^aaar  aodaX  *o«rraat*«aanodaX*  eoauapt  'd2H 
('d2  (add-»to-aaac"'oodaX  •e«rraat-aaa«BedaX*  ooaoapt  'dlH) 

(paateaa  ooooapt  ( ooaoapt a^aiylatead  *«oacaot-'aaaaMOdaX*>  >  > 

(dafaa  TKLX^OdMdCOKL  r3PCTI0P-DgIJa—>  (faaetioa) 

;;aftar  a  fuaotioa  la  applatead  wa  pXaoa  ttet  faaetloa  te  tte  aaor  aodaX  at 
;iXooaX  d2r  aaXoaa  it  ia  aXraady  praaaat*  te  ttet  oaao  wa  opgrado  ita  looaX  to  dX 
;;«tea  tte  fuaotioa  ia  aXraady  kaoaa  at  XaoaX  dir  «•  do  aotbite 
(oaao  (odr  (aaaoo  fuaotioa 

(fuaotioaa-fcaeoa  •oarraat  'uaaMcdaX*) ) ) 

<(aiX)  (add"to-uaar'^odaX  •ourraat'-uaoaaodaX*  fuaotioa  'dS)) 

('d2  (add-to-uaar*aodaX  ^niirraat  laararwlal'"  fuaotioa  *dl)>) 

(paateoo  fuaotioa  (fuaotioaa-aaplalaod  ♦oarraatt’-ttaamndal*^ )  I 
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Tnj«-uaiMooKb*nnjic-«oxAiaa>  (m«) 
nmttmx  «  tulM  L»  •lylatnad  «•  plAO*  tk^t  vnLm  la  tka  omc  aodal  «t 
;;la«aX  <12,  aalaaa  It  i«  alraady  praaaat,  ta  tkat  oa—  «•  opgrada  ita  laaal  to  <12 
/inkaa  tka  rala  la  alraady  fcaoaa  at  laaaX  dl,  aa  do  aotkiaf 
(daaa  <adr  (aaaoo  nla 

(raAaa«»taoan  •oorraato«aai»odal«) ) ) 

((all)  (add*to*aaar  'aortal  *eanaat-ttaai»odal*  tala  'd2)) 

('d2  (add^o^aaar" aodal  ‘rnirranf  naanmrta) *•  nila  'dl))) 

(paahaaa  tula  (CTlaa*a»p1aiaart  «oarraat-ttaanMdalM ) ) 


(dafua  TBA^OddMOPEL-lCTlurmr-kCCE—  (atria^) 

Mfuaotioa  tkat  iafesma  tka  «aa«  aodal  tkat  tka  uaar  kaa  •alaotad  a  aooaa 
;;aaaaitiva  otojaot  la  tka  ooataat  of  aa  aiq»laaatloa  to  raqaaat  aooaoa 
;;dooMaat  iToaiaar  iafoflaatlea  oa  tkat  ok^aot.  It  aota  ao  a  filtat  to 
;;rta»aralBa  If  tka  objaot  la  aimatktap  tkat  aaiata  la  oaz  rlnaaia  aedali  If 
;;ao  tka  oaar  aodal  la  iafoaad  tkat  aa  opdata  la  appreprlafta 
(lat  ((objaot  (lataca  (atzlaf-upoaaa  atriof)))) 

(ooad 

((flad  ooaoapt^by  ataa  objaot)  ;;objaot  la  a  ooaoapt 
(tall»uaaBanrtal-aoaaapt«kypartaat-acoaaa  objaot)) 

( (flnit^fnnntlna  lT^niaa  objaot)  ;;objaot  la  a  faaotloa 
(tall^Baazarwtal-fttaotioa-kypartaat^aooaaa  objaot) ) 

(t  til)  ;  f  objaot  aot  la  rtf  ala  aodal 

) ) )  ; ; foe  aoo  do  aotklad 

(dafua  TKLL-OSSIMaOCZi-oaKXPT-fYFCiaDCT-kOCXM  (ooaoapt) 

;/lf  a  uaar  aooaaaao  tka  do  am  oat  aaaaiaaz  do  of  aatatioa  aaaoolatad  wltk 
;;a  paztioular  ooaoapt  tkaa  «a  alao  add  tkat  ooaoapt  to  tkalz  uaar  aodal 
Wia  a  aiallar  aaaaar  to  «Aaa  tka  ooaoapt  ia  arpl.ataart  dlraotly 
(oaaa  (adz 

(aaaoo  ooaoapt  (ooaoapta^ltooot  *oufTaat*naaiaodBl*) ) ) 

((uU)  (add- to-uaar  aodal  ^ourraat-uaataiodal*  ooaoapt  'd2)) 

('d2  (add*to-qaar*  aodal  •ourraat-aaacaedal*  ooaoapt  'dl>) 

n 

(dafua  TSLXr-CSBMaDBX/-nnKno«-IYtKR7BXT-kCCBM  (fuaotloa) 

;;1Dm  tka  prarloua  fuaotloa  but  tka  kypartart  aooaaa  aooaaa  la  for  a  fuaotloa 
(oaaa  (odr 

(aaaau  fuaotloa  (fuaotloaa^kaoua  *<mrraat-uaamodal*>)) 

((all)  (add'  to  uaur-iMdul  •<mrraat-uaaxaodal*  fuaotloa  'd2)) 

('d2  (add-to-uaar'aodal  * ourr aat"»u aarandal*  fuaotloa  'dl>) 

)) 


(dafua  tSLb-OSDMODCL-Rn.l->«fkTOd-caJdRD  (rula  atatua) 

; }  If  tka  uaar  aakaa  uaa  of  tka  oapablUty  la  tka  U»C  latarf aaa  to  apoeif y  tka 
;;dafault  aotloa  takaa  for  a  rula  (a.p*,  alaaya  aMarnita,  tmzu*-off,  pra^t)  tkaa 
;;wa  lafar  tkay  ka^  a  aoaa  uadarataadlap  of  tkat  rula;  ita  loaal  la 
;itka  uaar  aidal  ia  aat  to  42. 


<okaapo*culo-atatoa*for-uaar  *ousToat>qaoaaodal*  rula  :atata  atatua  )) 


(dafua  TKUi-qiTkigOBL-IBPLK  (yMm/MI  IDOID  (rula) 

; ; if  tka  uaar  kaa  aawatklap  to  aay  about  a  rula  tkat  tkay  add  to  tko  ayatao 
;;doofootatloa  for  tkat  rula  (uaiap  tka  *add  rula  ocaaaat*  oapablUty  la 
;/lLC*a  latarfaoa),  tkaa  ua  lafar  tkat  tkay  kaua  a  aapklatlaatad  — 
not  tkat  rulai  ita  laaal  la  tka  uaar  aodal  la  aot  to  dl 
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nn  osxucT  Mcncos 

••ooBd  of  mothodo  oro  tlM  iAdlroot  mthodo«  Thooo 
;i;aot«  ohoago^  to  tho  oMr  aodoX  «ad  um  that  lafotaat  lea  to 
; ; ;  iaf or  nhanijat  to  ottec  ob joets  ia  tho  aoor  oodol .  Thoy  pcogogoto 
;;;oay  ohoagoo  throagh  tho  aodol  by  uoiag  thot  ohoago  oo  o 
/;;oao  oad  aolrlwg  uoo  of  *dopoadoot-oa*  la  tho  ifaMaii  aodol, 

;;;7hooo  aothodo  oro  iaploaoatod  ••  orooad  aothod^  oa  apdotoo  to  tho 
; ; ; *kaowXodgo*  aloto  (ruloo,  fuaotloao  oad  ooaoopto  kaooa)  ia  tho  ooor  aodol. 

(doteothod  (ssrr  raloo-kaooa)  ibofozo  (aow-ruloo-kaowa  (moor  poor  aodol) ) 

;;Thio  aothod  lapi  f  nto  tho  aotioa  thot  oay  nilo  whleh  a  moor  tnnira  laplloo 
;;thot  thoy  oloo  tho  foaotloao  oad  ooaoopto  oadorlylag  thot  ralo,  if  tho 

;;loool  of  kaewlodgo  lo  *dl"  thoa  tho  kaowlodgo  loool  of  both  oadorlylag  fuaotloao 

;;oad  eoaeopto  io  "df*;  if  tho  rulo  kaooiodgo  loool  io  *d2*  thoa  oaly  uadorlyiag  foaotloao 
;;ozo  laforzod  to  bo  at  loool  *d2*  oad  aothlag  lo  laforrod  obeat  ooaoopto  for  now 
(lot*  ( (ralo-ooop  (flad^apdoto 

(nloo*kao«a  oooz)  ao«^n>loo»>taooa)) 

(rulo  (ear  rulo-ooaon 
(looal  (odr  rulo-oeao>) 

(rulo-iaotoaoo  (flad-mlo»byaaoo  rulo)) 

(fuaotloao^llot  (fuaotloao  rulo^iaotoaoo) ) 

( ooaoopto-  llot 

(f iad*o  1 1- dopondoiit-oo-oooaopto  talo-laotoawo> ) 

(ao«^Ioool  'd3)) 

(If  (og  loool  'dl) 

;;  ohoago  aodol  to  rofloot  *d2*  kaowlodgo  loool  for  ooaoopto  uadorlyiag  thlo  rulo 
(dollot  (oooh-ooaoopt  ooaoopto-llot> 

(odd-to-uoor  aodol  *ourroat-uaoiBOdol* 

(hoao  oooh-ooaoopt)  noi^lofool)  > ) 

it  tot  both  kaoolodgo  looolo  *dl*  oad  *d2*  for  thlo  rulo  ia  tho  aoor  aodol 
;;oot  haowlodgo  loool  for  uadorlyiag  foaotloao  to  *d2* 

(dollot  (ooch-fuaetloa  fuaotloao— llot) 

(odd-t^ttoor  aodol  *ettrroat-uooxaodol*  ooeh-fnaetloa  now-lovol) ) ) ) 


(dofaothod  (SRT  fuaotloeo-kaown}  sboforo  (aow-fuaotloao-kaowa  (noor  uoor-aodol) ) 

/^fhlo  aothod  lo  oiallor  to  tho  provlouo  oao,  ■ao«pt  It  mao  aftor  updotoo  to  tho 
;;foaotloao  kaooa  olot  ia  tho  uoor  aodol.  It  iaplMMoto  ladlzoot  aothodo  that  allow 
;;uo  to  oot  tho  haowlodgo  lowol  for  ooaoopto  a  glwoa  fuaotloa  lo  dopoadoat  on  to 
n*dl*  if  tho  foaotloa'o  haowlodgo  lowol  hoo  gooo  to  *dl*  aad  *da*  If  tho  fuaotloa' o 
;;haowlodgo  lowol  hoo  gooo  to  *df* 

(lot*  ( (fuootioo-ooao  (flad-updoto 

(fuBCtloao-hoowa  uoor)  aoir^fmootloao-haowa) ) 
(fun-ooao  (cor  fuaotloo-ooao)) 

(lowol  (odr  f uactloa-ooao) ) 

(ooaoopto- liot  (flnd-oll-dopoadoBt'oo  ooaoopto 

(flad-fuBotloa-hyugao  fua-aoao))) 

<ao«^ lowol  <lf  (oq  lowol  'dl)  'dl  'd2))) 

(dollot  (oooh-ooaoopt  ooaoopto-llot) 

<odd-to*uoor  aodol  *curroat-uoozaodol* 

.aoM  oaeh-ooaooptt  bo<^  lowol) ) ) ) 


(doteothod  (SKTr  ooaoopto-haowa)  :boforo  (aoi^ooooopto-haowa  (aoor  uoor  aodol) ) 

;;fhlo  aothod  io  for  ooaooptu.  xt  uooo  tho  oa  t^dato  to  tho  *ooooopto-haowB*  slot  ia 
;;tho  uoor  aodol  to  iafor  thot  dopoadoot-oa  ooaoopto  oro  oloo  haowa  at  tho  »tmo  lowol 
(lot*  ( (ooaoopt-ooao  (flad-updoto 

(ooooopto-haowa  uoor)  aow  ooooopto-haowa) ) 

(ooaoopt  (eor  ooaoopt-ooao)) 

(lowol  (odr  ooaoopt -oooo) ) 

(ooaoopto-liot  (f iad-oll  ■dopaodoot-ce-  ncaoopto  ;;llot  of  laotoaeoo 

(flad-oeaoopt-by-aoao  ooaoopt))))  ;;of  thooo  ooaoopto 

(If  (og  lowol  'dl) 

;;lf  o  ooaoopt’  bolooga  to  o  uoor'o  *dl*  thoo  Ito  dopoadoat  -oa  eoaoopta 
;;oloo  boloag  ia  thlo  uoor'o  *di* 

(dollot  (oaeh-ooacopt  ooaoopto-llot) 

(odd-t»-uoor  aodol  *ourroot-oooiaodol* 

(aoao  oooh-ooaoopt)  'dl)) 

;;aa  tho  othor  hood  if  o  ooaoopto  boloogo  to  o  ooor'o  "df*  thoa  wa  iafor  toot 
;;dapoadoat-oo  ooaoopto  oro  haowo  ot  lowol  *d2*  or  bottor  —  ooaoopto 
;  /  o3  roody  la  tho  aodol  at  *df*  go  to  *dl*  thooo  obooot  oro  oddod  ot  lowol  "df* 
(dollot  (ooeh-ooooopt  ooooopto-llot) 

(lot  ( (thlo-ooaoopt  (Boao  oooh-ooaoopt))) 

(If  (Bull  (hoi^wall-dooo-uoor-haow  *oarroot-uoofBodol*  thlo-ooaoopt) ) 

(odd'  to-uoor  aodol  •ourroat-uoooidol* 


thlo-ooaoopt  'dl) )))))) 
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The  following  portion  of  the  framework  for  the  acquisition  subcom¬ 
ponent  is  provided  so  that  the  other  three  categories  of  acquisition  techniques  (as 
described  in  Section  3-2)  could  be  integrated  into  the  user  modelling  component. 
Methods  implemented  in  this  dissertation  are  those  shown  above;  all  are  implicit 
acquisition  methods. 


stn  njkzxsTzcAL  moo* 

Mt  of  ootboda  usoa  lofoimatlOB  thmt  thm  abotlatloal 

modulo  ooeamulotoo  for  tho  ooor.  Tboy  uao  tho  atotiatiool 

;;}dobo  to  lAfor  apoolfio  chan^a  or  addltlooa  to  tbo  aaor  modmX. 

/;  jThaao  oco  loooaplato  oa4  would  roqulro  o  ai^nlfleoftt  of  fort 
; ; ;  "oaolyriad*  tbo  atotlatiool  lAfoxmotiom  to  aoo  oaoatly  wdot 
;;iiuforouoo  It  abould  tri99or« 

(dofum  rtMCTXQIKOSBD  (fuaotioo)  (ifuoro  f uuotAoo) ) 
nn  TOTOdiw  McnoM 

;;;Aia  not  of  oothoda  ia  tbo  iatorfooo  uitk  o  tutorlod  ofpooont 
;;;for  updotlug  tbo  modal  ooatoata.  Jubjoota  OB  wbleh  tbo  uaor 
;;;boa  rmoolumd  apoclfio  tutotlaf  apiaodoa  oitbos  «t  tbolr  roquoat 
;;;or  a«  tbo  roault  of  auggoatlooa  from  izsP^Crltlo  vill  too  aaod 
;;;to  lafor  kmowloddo  lowala  ia  tbo  uaor  modol 

(dofaa  WmBi  OMCiay^'AmOMP  (oomoavt)  (l^morm  oeooopt) 

;;aay  ooooopt  oo  vbiob  a  uaor  boa  boao  tutorod  will  too  aot  to  louol  'd2 

) 

(dafuB  TCA-qui  i>aoKL-ft>»CTXCi»-'WrmMP  (fuaotloa)  (l^aoro  fuaotioo) 

;;«uiy  fuaotiou  oo  vbiob  a  uaor  baa  booo  tutorod  will  bo  aot  to  loool  'd2 

» 

;;;;  nnxcZT  MEncM 

///Ttola  aot  of  matboda  uaoa  oapUait  aoqulaitloo  toohai^iuoa  to  oitbor 
;;yoatabIiab  aa  ialtlol  uaor  modal  or  to  iatoroetlToly  quory  tbo  uaor 
;;;ifbao  tbo  ayatam  aooda  additiooal  iafotmatioo  or  olorifloatiou 
;i;ro9ordiaq  tbo  kaowlodqo  atoto  of  tbo  uaor. 

<dof«B  baK«t7f»»AtoCgy-Tiii»-nioiiXipq«  n 

;;aot  of  iatorootioa  quorioa  that  quoatioo  uaor  about  tboir  oj^ortiao 
) 


APPENDIX  D 


ACCESS  METHODS  IN  THE  USER  MODELLING  COMPONENT 

This  ^jpendix  shows  the  set  of  access  functions  that  are  available  for  the 
user  modelling  component  itself  as  well  as  for  other  system  components  to  “inter¬ 
rogate"  the  contents  of  model  instances,  llie  scheme  for  inplementing  access  is 
to  provide  general  accessor  functions  to  other  system  components  which  when 
called  cause  the  user  modelling  con^nent  to  invoke  the  qjpropriate  CLOS  method 
on  the  user  model  instance  representing  the  programmer  presently  using  the  sys¬ 
tem  —  that  instance  is  bound  to  Ae  global  variable  *current'Usermodel*. 


ft n  roMDMiarzAL  mbtsom 

t  tt  t  Wntbodg  th«t  AT*  a««d  to  acooos  tbo  oaor  modml  by  otbor  ayotos 
;;;;  oaopoMato  and  pvoooaaoa.  Tboaa  aro  aoMtiaao  rofarrad  to  aa- 
it  ft  tte  a&lToaaad  oafehoda  tbmy  ooataiA  ao  kaovlodya  aboot'  boo 
tttj  to  modify  or  updato  tha  «aor  modal. 

(dateaithod  mXCI-OOU-OUR-nPOVT  ((maor  aaaa  modaU.  Hap 'objacta^ 

it  naturaa  a  Hat  of  tboaa  ooaoapta,  mlaa  or  fuaotleaa  that  a  aaar  taoaa 

;;  that  ara  ia  tba  aryiaiant  Hat  *llap*ob^aota* 

fooad 

( (fia<Hooaoapt"bynMa  (oar  liap-objaotaH 
(iataraaotloa  liap^objaota  <astraot«>aMaa. 

(ooaoapta^taooa  «aar> ) > ) 

( (<iad*faaotloa«hy naraa  (oar  Xiap-ob)aotaM 
(iataraaotloa  ILap-objaeta  (astraot'-aoMoa 

(faaotioaa-kaooa  aaar)))) 

(t  (iataraaotloa  llap-^jaota  (axtraot^aaMo 

(ralaa^kaoaa  oaar)))))) 

(dateathod  imxci»POE<"<yCTIl-»OT-»PlOW?  ((oaar  aaar  modal)  Hap  objaota) 

;;JlatarBa  a  Hat  of  thoaa  ooaoapta,  nlaa,.  or  faaotioaa  that  a  aaar  OOU  POT  mow 
//that  ara  ia  tha  argmaaat  Hat  *Hap-^)aota* 


( (fiad-ooaoapt-by»wama  (car  Hap-objacta)) 

(aat-dlXfaraaoa  Hap-objacta  (aatraot^aamaa 

(ooaoapta-haowa  «aar))>) 

( (fiad*faaotioa*b]^aaaa  (car  liap-ob^aota)) 

(aat-dlf  faraaoa  Hap  'abjacta  (aatraot-aiMa 

(fuaotioaa-kaoaa  aaar) ) ) ) 
(t  (aat-dif faraaoa  Hap-mbjaota  (aKtraot-maMoa 

(ralao«*ltaooa  aaar)))))) 
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(d»flMthod  mza-DOKS-osBR-iiav-iMW-wiU*?  {(AMr  OMS-Mtei) 

;;rrai  tb«  AvgwMat  aitbo^a  rvtuxM  uy  oomomfit,  fuMtlo*  or 

;;orltlo  r«lo  tbot  tbo  ooor  door  Mt  loam  ot  loool  41 


ixioorpC  hy  ni—  <eor  llrp-«tojooto>) 

(■ojw  #'  (loab4o  (oottoapC) 

(uaXoor  <oq  (bnirwll  rhra  THtr  Irnmi  *e«rro&t'' 
(list  OOCkOOpt))) 

Xisp-ob  joota ) ) 

( (fio4*ftaotioo-by*n«M  (oox  Ii«p*>objoeto|) 

Coposn  #*  (iMihrts  (ftaotlool 

(«tloo«  (oq  <boo  —11-40— tsor“la>oo  •oBrx«ot-< 
(list  fttootiooH) 
lisp-objootan 

(t  (ospnsB  I*  (Isabdo  (los-nlo) 

(ualoss  (oq  (boif  soil  4oos-iissr*fcooo  *otrroot-< 
(list  Lor-r«ls))> 
lisp-objoots) ) 


I*  ftaetieo)  '41) 


itussiMKlsl*  lor»s«lo)  '41) 


(tectn  yotZCJ-OCSK-OOBS-VOT-nOV  (list-of-topios) 

;;riltors  tbo  topios  (oooospts)  iji  list*o<*t^ios  throoqb  tbo  ssor  of>4sl  sod 
;ir«tax»s  tboso  tbo  osos  4oos  sot  k»o«  st  sU 
(iftiob*-4oos««soc-ttOt*kaoo)  *o«rrsot-«soatf4ol*  list-of-topies)) 

(tefoa  BOMCT-nMU  (s-List) 

j /slots  io  usonodoli  oooosptSr  ftiBotioos,  sod  rolos-kaoim  sro  s^lists;  this  fttootieo  sstroots 
/ /tho  ASMS  (oss  ot  ssoh  itos)  fros  thst  list  so  nnsfisrliiMi  oporotioos  ooo  bo  porfdtMd 
(■spnsr  'oor  s-list)) 


(4of«o  ■Ol^iaeit'-OOU-K/mi-IMOV-nM  (objoets-list) 

//sfloopts  s  list  of  Lisr-OBJBCT4,  dstscmioss  ii  soob  ooo  is  Is  tbo  ssomortol  sod  sb  shot  Isool 
//tbs  osos  koows  tho  objoot.  rotsms  thro*  lists/  objoets  Inwwm  st  lovols  40^  dl  sod  42» 

(Isft  ((41-iist  nil) 

(42*list  ail) 

(4d-list  ail)) 

(dolist  (objoot  objocts-list) 

(ooso  (boi^osll  Ooss^asor*fcooo  •eiurrsot^nuMuasodol*  objoot) 

(41  (ptsh  objoot  41- list)) 

(43  (p«sb  objoot  42- list)) 

(40  (posh  objoot  dO-list)))> 

(list  (list  '40  40-list)  (list  '42  43-Ust)  (list  '41  4l-list)))) 

(^AMtbod  OOV-HKLb-OOKO-OdCK-aoV  ((osor  osor -00401)  lisp-ebjoot-asM) 

//sotmsos  tbo  *4*  loosl  of  tbo  srqvMot  *lisp- objoot* 

(oood 

((odr  (assoo  lisp-objoot-aano  (ooooopts-fcsooa  asor)))) 

((odr  (ssooe  lisp-objoot-asMo  (foaotioos-ksoos  asor)))) 

((odr  (osooo  lisp-objoot-asM  (ralos-ksoos  asor)))) 

(t  '40)))  ;;*40*  sooas  oot  kaows  st  oil 


(doteotbod  ADO-TO-onK-MOOKL  ( (asor  asor  ■odsl)  lisp-objoot  loool) 

;;  Adds  srqaMSt  lisp-objoot  to  sppreprioto  slot  is  tbis  isdiaidasls  asor  oodol 


( (fis^ooouopt  by  SMO  lisp-objoot)  /;tbio  is  a  LZfP  ooeoopt 

;/if  o  sow  oooospt  for  tbis  asor  tbos  odd  it  to  tbo  eosoopts-kBOwa  slot  is  tbo  asor  sodol 
(if  (Ball  (assoo  lisp-objoot  (aoBoopts-kacoB  asor))) 

(sotf  (ooooopts-kaowa  asor) 

(aoeoo  lisp-objoot  loool  (oaooopts-ksooa  asor) )  > 

;;olso  roplaoo  tbo  earroot  looal  vltb  tbo  sow  loaol  *  do  tbis  o^s  if  tboy  aro  tbo  ssoo 
(aaloss  (oq  '41  (odr  (assoo  lisp-objoot  (ooBoopts-kao—  asor>>)> 

(rplood  (assoo  lisp-objoot  (ooooopts-kaowa  asor))  lowol))) 

) 

( (fiad-faaotioo-by-Bsoo  lisp-objoot)  //tbis  is  a  LZAP  faBOtioo 

//if  a  aov  foaotioos  for  tbis  asor,  add  It  to  faaot Icos-kBOWB  slot  la  tbo  asor  aodol 
(if  (Ball  (assoo  lisp-objoot  (faaot ioas-kaowa  asor) ) ) 

(sotf  (faaot ioas-kaowa  asor) 

(BOOM  lisp-objoot  lowol  (faaotioas-kaowa  asor) ) ) 

//also  roplaoo  tho  oarroat  lowol  with  tbo  sow  lowsl  -  do  tbis  owaa  if  tboy  aro  tbo  smbo 
(aaloss  (oq  '41  (odr  (assoo  lisp  objoot  ( faaot  ioas-kaowa  asor)))) 

(rplood  (assoo  lisp-objoot  (faaotioas-kaowa  asor))  lowol))) 
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(t  ;  tdmf  mnXt  qm«  •  it«  m  nil« 

;;lf  •  M«  rul«  foe  «mz#  odd  it  to  ntioo*koo«f»  alot  la  thm  ooor  aodol 

(■If  (anil  (aaaeo  liap-^^oot  (raloa-lcaowa  oaor))! 

(Mtf  (roloo-lcaowa  war) 

(aooao  Uap-objoot  loool  (raloa^^kaoim  nsos) ) ) 

;;ols«  roplaoo  tko  oarroAw  iaool  vltfr  oott  loool  -  do  t&ia  ovoa  if  thoy  aro  tte 
(oalooa  (oq  'dl  (odr  (aaooe  llap-objoot  (galoo-kaooa  aaor)n) 

(xplaod  (asaoo  liap-ebjoot  (nUaa-kaowa  uaor) )  lovol) ) ) 

) 

)) 

(rtofaothod  AXJUUDY-BOIAXVDY  ( <tta«r  uaor  aodoU  liap-«tojoot-MM^ 

;;  Rotaxaa  trao  if  thm  arqoMat  *liap-objoot*  haa  booa  ajylaiaod  to  tbo  aaor 

;;  la  tao  paat  otaarvlao  eotaxaa  all 

(ooad 

( (fiad"Ooaoapt,«by-iiaoa  llap-oto joot-aaao) 
f aa»or  liap-cbjoot-aMo  (ooaoopta-a^lalaod  uaor))) 

( ( f  1  uit~f Tin  nt  1  on  tmm  llap*objoot>aaM) 

(aaMhar  Uap-^joet-uana  (fuaotloaa^ojylalaod  uaor))) 

(t 

taaohor  Ilap*otojoot-aaM  (ruloa*oi9lalaod  uaor))) 

)) 


(doteotaod  caAME^RJLX-msOd-rollHJfn  <  (uaor  uaor  oodal)  rulo  4koy  atato) 

;;  Chaagoa  tho  atato  of  tao  argoMot  *rulo*  dopoadiag  ea  tao  uolao  of  axgvMot  ■atato* 

;)  by  updating  tbo  approprlato  alota  —  probably  ahould  bo  ualag  a  alaglo  alot  vita  aa  o-Uat 

(eoao  atato 

( I alwaya-aeoopt 

)/  onahlo  tala  rulo  by  r— laalng  It  fra»  tbo  Hat  of  ruloa  tnraod  off  or  proaptod  for 
(If  (oot  (oMibor  rulo  (dafault-aooapt^ruloa  uaor))) 

(puaa  rulo  (dofault-aooopt-ruloa  uaor) ) ) 

(aotf  (ruloa-dlaablod  uaor) 

(doloto  rulo  (ruloa-dlaablod  uaor))) 

(aotf  (proBpt-oo-ruloa  uaor) 

(doloto  rulo  (proaipt^oo-ruloa  uaor) ) ) ) 

( lalooya'-ro^oet 

;;  dlaablo  taia  rulo*  add  it  to  tbo  Hat  of  ruloa  turaod  off,  rinor  It  fr«n  otaora 
(if  (not  Coabar  rulo  (ruloa«Nilaablod  uaor))) 

;)bat  oaly  if  it  la  not  alroody  taoro 
(puaa  rulo  (ruloa-dlaablod  uaor))) 

(aotf  (preapt*nMr> ruloa  uaor) 

(doloto  rulo  (pmoptTuloa  uaor) ) ) 

(aotf  (dofault-'ocoopt-ruloa  uaor) 

(doloto  rulo  (dafaolt^ucoopt-ruloa  uaor) ) ) ) 

(ipruopt-M 

/fodd  tala  rulo  to  Hat  of  thoao  waioh  ayuti  vlH  aak  tbo  uaor  about 
(if  (not  (Mobor  rulo  (pmtrr-oa-m  1  oa  uaor))) 

;;bot  ooly  if  It  la  aot  alroody  oo  that  Hat 
(puab  rulo  (prqupt"0^ ruloa  uaor))) 

(aotf  (ruloa'-diaoblod  uaor) 

(doloto  rulo  (ruloo*diaoblod  uaor) ) ) 

(aotf  (dofoult*oooapt*ruloa  uaor) 

(doloto  rulo  (dofoult^uooopt^ruloa  uaor) ) ) ) 

)) 

;;r;  Hotboda  tbot  prorlda  a  poralatoot  oopoblHty  to  tbo  uaor  uodon lag 
;;/)  ooipoooot.  Tboao  ai^ov  tbo  uaor  oodnl  to  bo  aoaod  to  aod 
;;;;  rotrioood  frm  a  filo  ao  that  It  coo  bo  rouaod  aad  Itoratluoly 
;;;;  oobaaood  duriag  aubaoguoat  uaoa  of  tXSP-Critrlo . 

(doteotbod  «AVB-ObJ»CT  ( (aolf  aooo-mlaia)  loptloaol  atroan) 

///■otbod  oo  tbo  olala  oloaa  to  aooo  aa  ladivldual  aodal  to  a  filo  tbot  ooa 
;;;ooa  bo  loodod  during  tbo  aoxt  aoaaioo  vltb  tbo  ayatao,  Tbla  aotuolly  oorlw 
;;;by  oroatiag  oodo  oad  putting  it  into  tbnt  filo  ae  tbot  wboo  tbo  filo  la  loodod 
;;;it  vHl  notco  aa  iaatonoo  of  tbo  uaor  uodoi  In  tbo  loool  aoulrooBoot.  Tbnt 
; ) ;  Inatonoo  oootoina  tbo  infoxBotloo  about  tbo  unor  bullt*up  ooor  tlao. 

(lot  ((Initorga  ail) 

(otbor-inita  all) 

(oloaa  (eloaa^f  aolf) ) ) 

(vltb-alota  (uaM)  aolf 

(dollat  (alot  (poll toloaa-a lota  oloaa)) 

(lot  ((alot^auMo  (polx talotd^aat  alot))) 

(ubao  (alot-boundp  aolf  alot-maao) 

(lot  ((voluo  (alot-ualuo  aolf  alot-'oado)) 

(laltorg  (oor  (poll lalotd^loitorga  alot)))) 

(if  initorg 

(aotf  initorga  *(',ioitorg  ',uuluo  .  ,ioitorga)) 
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(p««4  (Il«t  alot^aaM  valM)  otter-iaita) ) ) )  H 

(print  '  (aatf  (««t  ',aaM  'iaataao*) 

;>  (typ»"of  a«lf)  doaa  not  aork  pvoparly  la  PCb 
;;  OM  (olaaa-aaM  (olaa»-oe  —It)}  laatMMl 
(aalf  Inat aaoa  * ,  (olaaa^aaM*  elaaa> 

,|ialtar«a)) 


atr«M) 

(dollat  (ialt  ottar-ialta) 

(print  '(antf  (alot-anlun  (^at  ',naiM  'inatanon>  '^(flrat  Inlt)) 
• , (anoond  inlt)) 
atrMB) ) ) ) ) 


(dntentkod  axvt-qsSMiaDBL-IO-riLK  ((unnr  aaar««odnl)) 
n  —  nn  laagn  of  ttn  naar'a  aodnl  In  kia  dlmetory  ao  tknt  It  onn  bn  uand  thn 
;;  annt  tlan  that  LZbP-Critla  la  innotnd  **  inpl^anta  tbn  pnralatnnt  nanr  nodnl 
(Int  ((*pmekMqm^  (flnd-paokndn  'tiaM>)> 

(«rltb-opnn-flln  (atraan  (nar^n-patkaaMaa  (uanr-'dlmotory  oanr)  *tlo*aanmn«lal  .llap.nnnaaf ) 
zdlmotlon  toatpat 
I  If-daaa-aot-aalat  taraata) 

(faauit  "*• — —  *;;;  Modat  LZO)  Syntaat  rijaia  llap/  Baaat  10;  kaoliadai  TCMS  •>«-*) 
(aaaa-'objaet  uaar  atraaa)))) 

(dafan  LOk&-Od««*t0OlL»niOM  riLK  0 

;;  ratrlaaaa  frea  tka  uaar' a  iyabollca  heaa  dlraetory  kXa  uaar  nodal  for  uaa  by  LXdP  Crltrla 
(load  (aar^a-parkiiaaaa  (fasuaar*k«Mdlr)  *slo-aaamodal.llap.naaaat*) 
t If *doaa*not«aalat  all)) 


(dafan  CKCMC-m^-OdCII-MSOCL  () 

;  t  taa  tkla  fonotlon  to  Inatantlata  a  aaw  «aar  nodal  wkaa  oaa  doaa  not  alxaady  axlat 
; ;  It  can  ba  aiyandad  to  aaa  althar  a  ataadard  daf anlt  nodal  for  all  non  maara  or  to 
; ;  Inltlata  aspllelt  aoqttialtloa  by  tka  aaara  qpMatloaa  In  ordar  to  olaaalfy  kin  or 

; ;  kar  Into  a  ataraotypa  to  uaa  aa  a  atartind  paint  —  for  now  va-  will  uaa  an  initial 
;/  dafanlt  nodal  tkat  la  mpty 

(aatf  *carraiit>uaaxnodal«  («a)ra-(nr*^*^‘**^  ' oaar-nodal) ) ) 

(dafan  kCTZVAn^OSSR-MOOBL  0 

/;  tkla  function  la  protldad  to  tka  aaln  crltlo  aopina  to  altkar  ratrlaua  a  uaar  nodal 
;;  fr«n  a  flic  for  tkla  oaar  (ona  tkat  oaa  oraatad  dnrlnqi  prauloua  intaraotloaa  vltJt 
It  tka  ayatan)  or  oraata  a  dafault  for  a  flrat  tint  uaar. 

(If  (null  (load-aaar-nodal-fcon-flla)) 

(oraata-naw-aaar-nodal) 

(aatf  *oarraat*aaacaodal* 

(pot  (latarn 

(atrlap-upoaaa  rliuaar*ld)  'tuna)  ' Inataaoa) ) ) ) 

(dafan  TKLb-VdSmoOCL-CXZTZQdS-aXMZOd-CaiVLBR  () 

;;  Intarfaoa  fonotlon  tkat  la  uaad  to  Infoca  tka  oaainodal  tkat  tka  orltiqalnp  anniuaaat. 

;;  kaa  jaat  oaaplotad  a  aaaalon  vitk  tka  uaar  and  oartaln  *olaan*ap*  prooaaaaa  oaa  ba  ran 
(Inof  (tlnoa-inuokad  •oarraat-aaamodal*) ) 

(aatf  (data  updated  *oarrant*uaacnodal*) 

(tlnajprlnt-oarrant'^laa  all)) 

(lot*  ((ilo*alado«  (dot  xflnd  ■propran-uindoa  'rlei  sslo-^fxuna  zaalaotad-ok  t)) 

(propran  (aoliaaad  slo-vindoa  zpropran))) 

(aatf  (abon  old  ooda?  *oarraat-uaamodal*) 

(iloi irlo-fria-ato^^old  oodaz  propran)) 

(aatf  (aboir  nai^ooda?  *oarraat*aaacnedal*) 

(sloi  lalo-frana-akoa' Boa  oodaz  propran)) 

(datf  (akov^m^lanatloal  *«arraat*aaaaMdal*) 

(tint  uln  frana  aknir  oiplanatInnT  propran))) 

(aaaa  aaamodal^o-flla  *Ottrxaat«aaacnodal*) ) 


APPENDIX  E 


QUESTIONNAIRE  ON  USP 


Name: 

1 .  Which  programming  languages  axe  you  familiar  with? 

2.  In  which  of  the  above  language  are  you  the  most  proficient? 

3.  Please  provide  the  following  infonnation  about  previous  ex¬ 
perience  with  LISP: 

a.  Number  of  LISP  programs  you  have  written: 

b.  Approximate  lines  of  LISP  code  written  (circle  answer): 

•  None 

•  10  - 100 
•  100-1000 

•  1000-10,000 

•  over  10, (WO 

c.  Previous  formal  LISP  instruction  (circle  those  that  q^ly): 

•  Took  a  short  course  on  LISP. 

•  Introduced  to  LISP  in  a  Programming  Language 
Course 

•  Have  had  LISP  as  part  of  an  AI  course 

•  Received  individual  tutoring  on  USP 

•  Other. 

d.  Any  Self  Directed  instruction  on  LISP: 

•  Computer-based  instruction 

•  Books  used  on  my  own  to  study  LISP  (which  ones) 
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4.  How  much  do  you  know  about  die  following  LISP  concepts? 

l=could  define 
2=  am  familiar  with 
3=  never  heard  of  it 


Symbolic  Expression 

Quote 

Functions 

Conditionals 

Variables 

Side  effects 

Scoping 

Property  Lists 

Lists 

Mapping 

Recursions 

Multi-value  return 

Cons  cell 

List  iteration 

Evaluation 

Trae  (non-nil) 

Nil 

Atom 

S.  How  much  do  you  know  about  die  following  LISP  functions? 

1=  could  probably  write 

a  correct  expression  using  the  function, 

2=  am  aware  this  function  exists  but  need  help  with  its  syntax 
3=  have  heard  of  the  function  but  not  sure  what  it  does 

4=  never  heard  of  the  function  before 

COND 

LAMBDA 

DO 

APPLY 

DEFUN 

FORMAT 

MAPCAR 

EQUAL 

LET 

NULL 

EVAL 

NCONC 

MEMBER 

SETF 

SYMBOLP 

LET* 

CAR 

CASE 

NTH 

AND 

CONS 

ELT 

LIST 

INTERN 

NCONC 

ASSOC 

APPEND 

PROG 

AND 

STRING 

PRINT 

READ 

INDEX 


Acquisition  methods  100 
ACTIVIST  37 
Adaptation  31 

Advisory-dialog  systems  50, 182 
Analogy,  as  explaiutdon  approach 
116 

Analytical  critiquing  29 
Argumentation  117 
ATTENDING  system  34 

Bug  models  of  students  47 

CAI  16,186 
Canned  text  109,124 
Classification  approaches  53 
Coaching  14, 17, 33 
CODE  IMPROVER  65 
Collaborative  problem  solving  in 
humans  10, 175 

Common  LISP  Objects  System  95 
Communications  breakdowns  8 
Computer  advisors  26 
Concept-set,  used  for  explanations 
118 

Conceptual  graphs  94 
Conceptual  structuring  of  domain 
88 

Constituency  scheme  115 
Constraints  26 

Cooperative  problem  solving  2, 4, 6, 
11,14,22,49, 64, 107, 192 
Correlation  of  user  model  contents 
166, 167 
Critics  21 

Critics,  as  learning  environments  15 
Critics,  as  used  in  planning  systems 
26 

Critics,  educatiorutl  30 


Critics,  performance  30 
Critiquing  24, 109, 183, 190 
Critiquing  process  27 
Critiquing  strategies  29 
CRITTER  35 

Decision  making  in  critics  36 
Decision  making,  knowledge-based 
support  of  6 

Decision  support  systems  4, 37 
DecisionLab  36 
Deep  domain  models  1 16 
Dependent-on  relationship  90 
Design  Advisor  35 
Dialog  3,8,19,49,133,146 
Differential  critiquing  29 
Differential  descriptions  120 
Diffi»rential  modelling  34,44 
Direct  acquisition  52, 180 
Direct  methods  139, 145, 146 
Discourse  comprehension  112 
Distributed  user  modelling  1 85 
Document  Examiner  105,110 
Domain  model  78, 87, 133, 152, 
186, 191 

Domain  model  entities  92 
Domain  model  relationships  92 
Donuiin  model,  layers  in  96 

EES  101,113 

Emotional  impact  of  critics  30 
Evaluation  of  user  model  156 
Evaluation  test  scenarios  159 
Expert  systems  5, 6 
Explanation  component  98 
Explanation  in  knowledge-based  sys 
terns  112 

Explanation  levels  118,119,192 
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Explanation,  meanings  of  106 
Explanation,  role  of  in  learning  19 
Explanation-giving  88 
E3q>lanations  as  cognitive  represen¬ 
tations  106 

Explanations  in  LlSP-Critic  68 
Explanations,  failures  of  1 10 
Explanations,  functions  for  108 
Explanations,  need  for  in  critics  32 
Explanations,  presentation  of  119 
Explicit  user  model  acquisition  59, 
176 

Exploratory  learning  16 

Fall-back  capability  113 

Generic  task  methodology  1 16 
Genetic  graphs  45, 59 
Goal  and  plan  recognition  8, 48 
Goal  recognition,  in  critics  27 
GRACE  38 
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